Merge pull request #1993 from rembo10/develop

Time for HP v0.5

.gitignore

@@ -64,3 +64,6 @@ _ReSharper*/
 /logs
 .project
 .pydevproject
+
+
+headphones_docs

.pep8

@@ -0,0 +1,17 @@
+[pep8]
+# E111 indentation is not a multiple of four
+# E121 continuation line under-indented for hanging indent
+# E122 continuation line missing indentation or outdented
+# E124 closing bracket does not match visual indentation
+# E125 continuation line with same indent as next logical line
+# E126 continuation line over-indented for hanging indent
+# E127 continuation line over-indented for visual indent
+# E128 continuation line under-indented for visual indent
+# E261 at least two spaces before inline comment
+# E262 inline comment should start with '# '
+# E265 block comment should start with '# '
+# E302 expected 2 blank lines, found 1
+# E501 line too long (312 > 160 characters)
+# E502 the backslash is redundant between brackets
+ignore = E111,E121,E122,E123,E124,E125,E126,E127,E128,E261,E262,E265,E302,E501,E502
+max-line-length = 160

.travis.yml

@@ -0,0 +1,20 @@
+# Travis CI configuration file
+# http://about.travis-ci.org/docs/
+
+language: python
+
+# Available Python versions:
+# http://about.travis-ci.org/docs/user/ci-environment/#Python-VM-images
+python:
+  - "2.6"
+  - "2.7"
+install:
+ - pip install pyOpenSSL
+ - pip install pylint
+ - pip install pyflakes
+ - pip install pep8
+script:
+  - pep8 headphones
+  - pylint --rcfile=pylintrc headphones
+  - pyflakes headphones
+  - nosetests headphones

API.md

@@ -0,0 +1,109 @@
+# API Reference
+The API is still pretty new and needs some serious cleaning up on the backend but should be reasonably functional. There are no error codes yet.
+
+## General structure
+The API endpoint is `http://ip:port + HTTP_ROOT + /api?apikey=$apikey&cmd=$command`
+
+Data response in JSON formatted. If executing a command like "delArtist" or "addArtist" you'll get back an "OK", else, you'll get the data you requested.
+
+## API methods
+
+### getIndex
+Fetch data from index page. Returns: ArtistName, ArtistSortName, ArtistID, Status, DateAdded, [LatestAlbum, ReleaseDate, AlbumID], HaveTracks, TotalTracks, IncludeExtras, LastUpdated, [ArtworkURL, ThumbURL]: a remote url to the artwork/thumbnail.
+
+To get the cached image path, see getArtistArt command. ThumbURL is added/updated when an artist is added/updated. If your using the database method to get the artwork, it's more reliable to use the ThumbURL than the ArtworkURL)
+
+### getArtist&id=$artistid
+Fetch artist data. returns the artist object (see above) and album info: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type, ArtworkURL: hosted image path. For cached image, see getAlbumArt command)
+
+### getAlbum&id=$albumid
+Fetch data from album page. Returns the album object, a description object and a tracks object. Tracks contain: AlbumASIN, AlbumTitle, TrackID, Format, TrackDuration (ms), ArtistName, TrackTitle, AlbumID, ArtistID, Location, TrackNumber, CleanName (stripped of punctuation /styling), BitRate)
+
+### getUpcoming
+Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type
+
+### getWanted
+Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type
+
+### getSimilar
+Returns similar artists  - with a higher "Count" being more likely to be similar. Returns: Count, ArtistName, ArtistID
+
+### getHistory
+Returns: Status, DateAdded, Title, URL (nzb), FolderName, AlbumID, Size (bytes)
+
+### getLogs
+Not working yet
+
+### findArtist&name=$artistname[&limit=$limit]
+Perform artist query on musicbrainz. Returns: url, score, name, uniquename (contains disambiguation info), id)
+
+### findAlbum&name=$albumname[&limit=$limit]
+Perform album query on musicbrainz. Returns: title, url (artist), id (artist), albumurl, albumid, score, uniquename (artist - with disambiguation)
+
+### addArtist&id=$artistid
+Add an artist to the db by artistid)
+
+### addAlbum&id=$releaseid
+Add an album to the db by album release id
+
+### delArtist&id=$artistid
+Delete artist from db by artistid)
+
+### pauseArtist&id=$artistid
+Pause an artist in db)
+
+### resumeArtist&id=$artistid
+Resume an artist in db)
+
+### refreshArtist&id=$artistid
+Refresh info for artist in db from musicbrainz
+
+### queueAlbum&id=$albumid[&new=True&lossless=True]
+Mark an album as wanted and start the searcher. Optional paramters: 'new' looks for new versions, 'lossless' looks only for lossless versions
+
+### unqueueAlbum&id=$albumid
+Unmark album as wanted / i.e. mark as skipped
+
+### forceSearch
+force search for wanted albums - not launched in a separate thread so it may take a bit to complete
+### forceProcess
+Force post process albums in download directory - also not launched in a separate thread
+
+### getVersion
+Returns some version information: git_path, install_type, current_version, installed_version, commits_behind
+
+### checkGithub
+Updates the version information above and returns getVersion data
+
+### shutdown
+Shut down headphones
+
+### restart
+Restart headphones
+
+### update
+Update headphones - you may want to check the install type in get version and not allow this if type==exe
+
+### getArtistArt&id=$artistid
+Returns either a relative path to the cached image, or a remote url if the image can't be saved to the cache dir
+
+getAlbumArt&id=$albumid
+see above
+
+### getArtistInfo&id=$artistid
+Returns Summary and Content, both formatted in html.
+
+### getAlbumInfo&id=$albumid
+See above, returns Summary and Content.
+
+### getArtistThumb&id=$artistid
+Returns either a relative path to the cached thumbnail artist image, or an http:// address if the cache dir can't be written to.
+
+### getAlbumThumb&id=$albumid
+See above.
+
+### choose_specific_download&id=$albumid
+Gives you a list of results from searcher.searchforalbum(). Basically runs a normal search, but rather than sorting them and downloading the best result, it dumps the data, which you can then pass on to download_specific_release(). Returns a list of dictionaries with params: title, size, url, provider & kind - all of these values must be passed back to download_specific_release
+
+### download_specific_release&id=albumid&title=$title&size=$size&url=$url&provider=$provider&kind=$kind
+Allows you to manually pass a choose_specific_download release back to searcher.send_to_downloader()

API_REFERENCE

@@ -1,71 +0,0 @@
-The API is still pretty new and needs some serious cleaning up on the backend but should be
-reasonably functional. There are no error codes yet, 
-
-
-General structure:
-http://localhost:8181 + HTTP_ROOT + /api?apikey=$apikey&cmd=$command
-
-Data returned in json format. If executing a command like "delArtist" or "addArtist" you'll get back an "OK", else, you'll get the data you requested
-
-$commands&parameters[&optionalparameters]:
-
-
-getIndex (fetch data from index page. Returns: ArtistName, ArtistSortName, ArtistID, Status, DateAdded,
-												[LatestAlbum, ReleaseDate, AlbumID], HaveTracks, TotalTracks,
-												IncludeExtras, LastUpdated, [ArtworkURL, ThumbURL]: a remote url to the artwork/thumbnail. To get the cached image path, see getArtistArt command.
-                                                                                                    ThumbURL is added/updated when an artist is added/updated. If your using the database method to get the artwork,
-                                                                                                    it's more reliable to use the ThumbURL than the ArtworkURL)
-
-getArtist&id=$artistid (fetch artist data. returns the artist object (see above) and album info: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type, ArtworkURL: hosted image path. For cached image, see getAlbumArt command)
-
-getAlbum&id=$albumid (fetch data from album page. Returns the album object, a description object and a tracks object. Tracks contain: AlbumASIN, AlbumTitle, TrackID, Format, TrackDuration (ms), ArtistName, TrackTitle, AlbumID, ArtistID, Location, TrackNumber, CleanName (stripped of punctuation /styling), BitRate)
-
-getUpcoming (Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type)
-
-getWanted (Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type)
-
-getSimilar (Returns similar artists  - with a higher "Count" being more likely to be similar. Returns: Count, ArtistName, ArtistID)
-
-getHistory (Returns: Status, DateAdded, Title, URL (nzb), FolderName, AlbumID, Size (bytes))
-
-getLogs (not working yet)
-
-findArtist&name=$artistname[&limit=$limit] (perform artist query on musicbrainz. Returns: url, score, name, uniquename (contains disambiguation info), id)
-
-findAlbum&name=$albumname[&limit=$limit]  (perform album query on musicbrainz. Returns: title, url (artist), id (artist), albumurl, albumid, score, uniquename (artist - with disambiguation)
-
-addArtist&id=$artistid (add an artist to the db by artistid)
-addAlbum&id=$releaseid (add an album to the db by album release id)
-
-delArtist&id=$artistid (delete artist from db by artistid)
-
-pauseArtist&id=$artistid (pause an artist in db)
-resumeArtist&id=$artistid (resume an artist in db)
-
-refreshArtist&id=$artistid (refresh info for artist in db from musicbrainz)
-
-queueAlbum&id=$albumid[&new=True&lossless=True] (Mark an album as wanted and start the searcher. Optional paramters: 'new' looks for new versions, 'lossless' looks only for lossless versions)
-unqueueAlbum&id=$albumid (Unmark album as wanted / i.e. mark as skipped)
-
-forceSearch (force search for wanted albums - not launched in a separate thread so it may take a bit to complete)
-forceProcess (force post process albums in download directory - also not launched in a separate thread)
-
-getVersion (Returns some version information: git_path, install_type, current_version, installed_version, commits_behind
-checkGithub (updates the version information above and returns getVersion data)
-
-shutdown (shut down headphones)
-restart (restart headphones)
-update (update headphones - you may want to check the install type in get version and not allow this if type==exe)
-
-getArtistArt&id=$artistid  (Returns either a relative path to the cached image, or a remote url if the image can't be saved to the cache dir)
-getAlbumArt&id=$albumid (see above)
-
-getArtistInfo&id=$artistid (Returns Summary and Content, both formatted in html)
-getAlbumInfo&id=$albumid (See above, returns Summary and Content)
-
-getArtistThumb&id=$artistid (Returns either a relative path to the cached thumbnail artist image, or an http:// address if the cache dir can't be written to)
-getAlbumThumb&id=$albumid (see above)
-
-choose_specific_download&id=$albumid (Gives you a list of results from searcher.searchforalbum(). Basically runs a normal search, but rather than sorting them and downloading the best result, it dumps the data, which you can then pass on to download_specific_release(). Returns a list of dictionaries with params: title, size, url, provider & kind - all of these values must be passed back to download_specific_release)
-
-download_specific_release&id=albumid&title=$title&size=$size&url=$url&provider=$provider&kind=$kind (Allows you to manually pass a choose_specific_download release back to searcher.send_to_downloader())

CHANGELOG

@@ -0,0 +1,7 @@
+v0.5 Released 10 Nov 2014
+-------------------------
+- Several bug fixes (please retest your posted issues)
+- Cue splitter
+- Other improvements
+
+ 

CONTRIBUTING.md

@@ -0,0 +1,35 @@
+# Contributing to Headphones
+
+## For users
+In case you read this because you are posting an issue, please take a minute and conside the things below. The issue tracker is not a support forum. It is primarily intended to submit bugs, improvements or feature requests. However, we are glad to help you, and make sure the problem is not caused by Headphones, but don't expect step-by-step answers.
+
+* Use the search function. Chances are that your problem is already discussed.
+* Visit the [Troubleshooting](../../wiki/TroubleShooting) wiki first.
+* Use [proper formatting](https://help.github.com/articles/github-flavored-markdown/). Paste your logs in code blocks.
+* Close your issue if you resolved it.
+
+## For developers
+If you think you can contribute code to the Headphones repository, do not hesitate to submit a pull request. 
+
+### Branches
+All pull requests should be based on the `develop` branch. When you want to develop a new feature, clone the repository with `git clone origin/develop -b FEATURE_NAME`. Use meaningful commit messages.
+
+### Code compatibility
+The code should work with Python 2.6 and 2.7. Note that Headphones runs on different platforms, including Network Attached Storage devices such as Synology.
+
+Re-use existing code. Do not hesitate to add logging in your code. You can the logger module `headphones.logger.*` for this. Web requests are invoked via `headphones.request.*` and derived ones. Use these methods to automatically add proper and meaningful error handling.
+
+### Code conventions
+Altough Headphones did not adapt a code convention in the past, we try to follow the [PEP8](http://legacy.python.org/dev/peps/pep-0008/) conventions for future code. A short summary to remind you (copied from http://wiki.ros.org/PyStyleGuide):
+
+ * 4 space indentation
+ * 80 characters per line
+ * `package_name`
+ * `ClassName`
+ * `method_name`
+ * `field_name`
+ * `_private_something`
+ * `self.__really_private_field`
+ * `_global`
+ 
+Document your code!

Headphones.py

@@ -14,15 +14,14 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os, sys
+import os
+import sys
 
 # Ensure lib added to path, before any other imports
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib/'))
 
 from headphones import webstart, logger
 
-from configobj import ConfigObj
-
 import locale
 import time
 import signal
@@ -33,6 +32,7 @@ import headphones
 signal.signal(signal.SIGINT, headphones.sig_handler)
 signal.signal(signal.SIGTERM, headphones.sig_handler)
 
+
 def main():
     """
     Headphones application entry point. Parses arguments, setups encoding and
@@ -63,16 +63,24 @@ def main():
         headphones.SYS_ENCODING = 'UTF-8'
 
     # Set up and gather command line arguments
-    parser = argparse.ArgumentParser(description='Music add-on for SABnzbd+, Transmission and more.')
+    parser = argparse.ArgumentParser(
+        description='Music add-on for SABnzbd+, Transmission and more.')
 
-    parser.add_argument('-v', '--verbose', action='store_true', help='Increase console logging verbosity')
+    parser.add_argument(
-    parser.add_argument('-q', '--quiet', action='store_true', help='Turn off console logging')
+        '-v', '--verbose', action='store_true', help='Increase console logging verbosity')
-    parser.add_argument('-d', '--daemon', action='store_true', help='Run as a daemon')
+    parser.add_argument(
-    parser.add_argument('-p', '--port', type=int, help='Force Headphones to run on a specified port')
+        '-q', '--quiet', action='store_true', help='Turn off console logging')
-    parser.add_argument('--datadir', help='Specify a directory where to store your data files')
+    parser.add_argument(
+        '-d', '--daemon', action='store_true', help='Run as a daemon')
+    parser.add_argument(
+        '-p', '--port', type=int, help='Force Headphones to run on a specified port')
+    parser.add_argument(
+        '--datadir', help='Specify a directory where to store your data files')
     parser.add_argument('--config', help='Specify a config file to use')
-    parser.add_argument('--nolaunch', action='store_true', help='Prevent browser from launching on startup')
+    parser.add_argument('--nolaunch', action='store_true',
-    parser.add_argument('--pidfile', help='Create a pid file (only relevant when running as a daemon)')
+                        help='Prevent browser from launching on startup')
+    parser.add_argument(
+        '--pidfile', help='Create a pid file (only relevant when running as a daemon)')
 
     args = parser.parse_args()
 
@@ -83,7 +91,8 @@ def main():
 
     if args.daemon:
         if sys.platform == 'win32':
-            sys.stderr.write("Daemonizing not supported under Windows, starting normally\n")
+            sys.stderr.write(
+                "Daemonizing not supported under Windows, starting normally\n")
         else:
             headphones.DAEMON = True
             headphones.QUIET = True
@@ -91,11 +100,14 @@ def main():
     if args.pidfile:
         headphones.PIDFILE = str(args.pidfile)
 
-        # If the pidfile already exists, headphones may still be running, so exit
+        # If the pidfile already exists, headphones may still be running, so
+        # exit
         if os.path.exists(headphones.PIDFILE):
-            sys.exit("PID file '" + headphones.PIDFILE + "' already exists. Exiting.")
+            sys.exit(
+                "PID file '" + headphones.PIDFILE + "' already exists. Exiting.")
 
-        # The pidfile is only useful in daemon mode, make sure we can write the file properly
+        # The pidfile is only useful in daemon mode, make sure we can write the
+        # file properly
         if headphones.DAEMON:
             headphones.CREATEPID = True
 
@@ -103,9 +115,11 @@ def main():
                 with open(headphones.PIDFILE, 'w') as fp:
                     fp.write("pid\n")
             except IOError as e:
-                raise SystemExit("Unable to write PID file: %s [%d]", e.strerror, e.errno)
+                raise SystemExit(
+                    "Unable to write PID file: %s [%d]", e.strerror, e.errno)
         else:
-            logger.warn("Not running in daemon mode. PID file creation disabled.")
+            logger.warn(
+                "Not running in daemon mode. PID file creation disabled.")
 
     # Determine which data directory and config file to use
     if args.datadir:
@@ -114,27 +128,28 @@ def main():
         headphones.DATA_DIR = headphones.PROG_DIR
 
     if args.config:
-        headphones.CONFIG_FILE = args.config
+        config_file = args.config
     else:
-        headphones.CONFIG_FILE = os.path.join(headphones.DATA_DIR, 'config.ini')
+        config_file = os.path.join(headphones.DATA_DIR, 'config.ini')
 
     # Try to create the DATA_DIR if it doesn't exist
     if not os.path.exists(headphones.DATA_DIR):
         try:
             os.makedirs(headphones.DATA_DIR)
         except OSError:
-            raise SystemExit('Could not create data directory: ' + headphones.DATA_DIR + '. Exiting....')
+            raise SystemExit(
+                'Could not create data directory: ' + headphones.DATA_DIR + '. Exiting....')
 
     # Make sure the DATA_DIR is writeable
     if not os.access(headphones.DATA_DIR, os.W_OK):
-        raise SystemExit('Cannot write to the data directory: ' + headphones.DATA_DIR + '. Exiting...')
+        raise SystemExit(
+            'Cannot write to the data directory: ' + headphones.DATA_DIR + '. Exiting...')
 
     # Put the database in the DATA_DIR
     headphones.DB_FILE = os.path.join(headphones.DATA_DIR, 'headphones.db')
-    headphones.CFG = ConfigObj(headphones.CONFIG_FILE, encoding='utf-8')
 
     # Read config and start logging
-    headphones.initialize()
+    headphones.initialize(config_file)
 
     if headphones.DAEMON:
         headphones.daemonize()
@@ -147,24 +162,25 @@ def main():
         http_port = args.port
         logger.info('Using forced web server port: %i', http_port)
     else:
-        http_port = int(headphones.HTTP_PORT)
+        http_port = int(headphones.CONFIG.HTTP_PORT)
 
     # Try to start the server. Will exit here is address is already in use.
-    webstart.initialize({
+    web_config = {
         'http_port': http_port,
-        'http_host': headphones.HTTP_HOST,
+        'http_host': headphones.CONFIG.HTTP_HOST,
-        'http_root': headphones.HTTP_ROOT,
+        'http_root': headphones.CONFIG.HTTP_ROOT,
-        'http_proxy': headphones.HTTP_PROXY,
+        'http_proxy': headphones.CONFIG.HTTP_PROXY,
-        'enable_https': headphones.ENABLE_HTTPS,
+        'enable_https': headphones.CONFIG.ENABLE_HTTPS,
-        'https_cert': headphones.HTTPS_CERT,
+        'https_cert': headphones.CONFIG.HTTPS_CERT,
-        'https_key': headphones.HTTPS_KEY,
+        'https_key': headphones.CONFIG.HTTPS_KEY,
-        'http_username': headphones.HTTP_USERNAME,
+        'http_username': headphones.CONFIG.HTTP_USERNAME,
-        'http_password': headphones.HTTP_PASSWORD,
+        'http_password': headphones.CONFIG.HTTP_PASSWORD,
-    })
+    }
+    webstart.initialize(web_config)
 
-    if headphones.LAUNCH_BROWSER and not args.nolaunch:
+    if headphones.CONFIG.LAUNCH_BROWSER and not args.nolaunch:
-        headphones.launch_browser(headphones.HTTP_HOST, http_port,
+        headphones.launch_browser(headphones.CONFIG.HTTP_HOST, http_port,
-            headphones.HTTP_ROOT)
+                                  headphones.CONFIG.HTTP_ROOT)
 
     # Start the background threads
     headphones.start()

README.md

@@ -1,17 +1,18 @@
-#![preview thumb](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png)Headphones
+#![Headphones Logo](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png) Headphones
 
-###Support & Discuss
+Headphones is an automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, µTorrent and Blackhole.
 
-You are free to join the HP support community on IRC where you can ask questions, hang around and discuss anything related to HP.
+## Support & Discuss
+You are free to join the Headphones support community on IRC where you can ask questions, hang around and discuss anything related to HP.
 
-1. Use any IRC client and connect to the Freenode server.
+1. Use any IRC client and connect to the Freenode server, `irc.freenode.net`.
-2. Join #headphones
+2. Join the `#headphones` channel.
 
-###Installation and Notes
+## Installation and Notes
 
-[Read our Wiki](../../wiki) on how to install and use HeadPhones properly.
+* [Installation page](../../wiki/Usage-guide) shows you how to install Headphones.
-
+* [Usage guide](../../wiki/Usage-guide) introduces you to Headphones.
-[**Troubleshooting** page](../../wiki/TroubleShooting) in the wiki can help you with comon problems.
+* [Troubleshooting page](../../wiki/TroubleShooting) in the wiki can help you with common problems.
 
 **Issues** can be reported on the GitHub issue tracker considering these rules:
 
@@ -28,16 +29,14 @@ You are free to join the HP support community on IRC where you can ask questions
 If you **comply with these rules** you can [post your request/issue](http://github.com/rembo10/headphones/issues).
 
 **Support** the project by implementing new features, solving support tickets and provide bug fixes.
-If you change something in the code always make a PR to the developer branch instead of the master branch.
 
+## Screenshots
 
-###Screenshots
+Homepage (Artist Overview):
-
-Homepage (Artist Overview)
 
 ![preview thumb](http://i.imgur.com/LZO9a.png)
 
-One of the many settings pages....
+One of the many settings pages:
 
 ![preview thumb](http://i.imgur.com/xcWNy.png)
 
@@ -49,7 +48,7 @@ Import Your Favorite Artists:
 
 ![preview thumb](http://i.imgur.com/6tZoC.png)
 
-Artist Search Results (also search by album!):
+Search Results:
 
 ![preview thumb](http://i.imgur.com/rIV0P.png)
 
@@ -61,5 +60,5 @@ Album Page with track overview:
 
 ![preview thumb](http://i.imgur.com/kcjES.png)
 
-
+## License
-This is free software under the GPL v3 open source license - so feel free to do with it what you wish.
+This is free software under the GPL v3 open source license. Feel free to do with it what you wish, but any modification must be open sourced. A copy of the license is included.

data/interfaces/default/artist.html

@@ -163,17 +163,17 @@
 	}
 
 	<%
-    if headphones.SONGKICK_FILTER_ENABLED:
+    if headphones.CONFIG.SONGKICK_FILTER_ENABLED:
         songkick_filter_enabled = "true"
     else:
         songkick_filter_enabled = "false"
 
-    if not headphones.SONGKICK_LOCATION:
+    if not headphones.CONFIG.SONGKICK_LOCATION:
     	songkick_location = "none"
     else:
-        songkick_location = headphones.SONGKICK_LOCATION
+        songkick_location = headphones.CONFIG.SONGKICK_LOCATION
 
-    if headphones.SONGKICK_ENABLED:
+    if headphones.CONFIG.SONGKICK_ENABLED:
         songkick_enabled = "true"
     else:
         songkick_enabled = "false"
@@ -186,7 +186,7 @@
 
 		template = '<li><a target="_blank" href="URI"><span class="sk-name">NAME</span><span class="sk-location">LOC</span></a></li>';
 
-		$.getJSON("https://api.songkick.com/api/3.0/artists/mbid:${artist['ArtistID']}/calendar.json?apikey=${headphones.SONGKICK_APIKEY}&jsoncallback=?",
+		$.getJSON("https://api.songkick.com/api/3.0/artists/mbid:${artist['ArtistID']}/calendar.json?apikey=${headphones.CONFIG.SONGKICK_APIKEY}&jsoncallback=?",
 			function(data){
 				if (data['resultsPage'].totalEntries >= 1) {
 
@@ -269,7 +269,7 @@
 			$('#dialog').dialog();
 			event.preventDefault();
 		});
-		$('#menu_link_modifyextra').click(function() {
+		$('#menu_link_modifyextra').click(function(event) {
 			$('#dialog').dialog();
 			event.preventDefault();
 		});

data/interfaces/default/base.html

@@ -38,7 +38,7 @@
 			</div>
 		% elif headphones.CURRENT_VERSION != headphones.LATEST_VERSION and headphones.COMMITS_BEHIND > 0 and headphones.INSTALL_TYPE != 'win':
 			<div id="updatebar">
-				A <a href="https://github.com/${headphones.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
+				A <a href="https://github.com/${headphones.CONFIG.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
 			</div>
 		% endif
 
@@ -96,8 +96,8 @@
 				%if version.HEADPHONES_VERSION != 'master':
 				(${version.HEADPHONES_VERSION})
 				%endif
-				%if headphones.GIT_BRANCH != 'master':
+				%if headphones.CONFIG.GIT_BRANCH != 'master':
-				(${headphones.GIT_BRANCH})
+				(${headphones.CONFIG.GIT_BRANCH})
 				%endif
 			</div>
 		</footer>
@@ -119,7 +119,7 @@
 	    $(document).ready(function() {
 	        $('form:first *:input[type!=hidden]:first').focus();
 	        try{
-	            var type = window.localStorage.getItem('search_type');
+	            var type = window.localStorage.getItem('search_type') || "artist";
 	            $("#search_type").val(type);
 	        } catch(e) {
 	        }

data/interfaces/default/config.html

@@ -50,13 +50,13 @@
                                     <label title="Username for web server authentication. Leave empty to disable.">
                                         HTTP Username
                                     </label>
-                                    <input type="text" name="http_username" value="${config['http_user']}" size="30">
+                                    <input type="text" name="http_username" value="${config['http_username']}" size="30">
                                 </div>
                                 <div class="row">
                                     <label title="Password for web server authentication. Leave empty to disable.">
                                         HTTP Password
                                     </label>
-                                    <input type="password" name="http_password" value="${config['http_pass']}" size="30">
+                                    <input type="password" name="http_password" value="${config['http_password']}" size="30">
                                 </div>
                                 <div class="row checkbox">
                                     <input type="checkbox" name="launch_browser" value="1" ${config['launch_browser']} />
@@ -86,7 +86,7 @@
                             <fieldset>
                                 <legend>API</legend>
                                 <div class="row checkbox">
-                                    <input type="checkbox" id="useapi" name="api_enabled" id="api_enabled" value="1" ${config['api_enabled']} />
+                                    <input type="checkbox" id="api_enabled" name="api_enabled" value="1" ${config['api_enabled']} />
                                     <label title="Allow remote applications to interface with Headphones">
                                         Enable API
                                     </label>
@@ -166,25 +166,25 @@
                                     <label title="SABnzbd username. Leave empty if not applicable.">
                                         SABnzbd Username
                                     </label>
-                                    <input type="text" name="sab_username" value="${config['sab_user']}" size="20">
+                                    <input type="text" name="sab_username" value="${config['sab_username']}" size="20">
                                 </div>
                                 <div class="row">
                                     <label title="SABnzbd password. Leave empty if not applicable.">
                                         SABnzbd Password
                                     </label>
-                                    <input type="password" name="sab_password" value="${config['sab_pass']}" size="20">
+                                    <input type="password" name="sab_password" value="${config['sab_password']}" size="20">
                                 </div>
                                 <div class="row">
                                     <label title="SABnzbd API key. Can be found in SABnzbd settings.">
                                         SABnzbd API key
                                     </label>
-                                    <input type="text" name="sab_apikey" value="${config['sab_api']}" size="36">
+                                    <input type="text" name="sab_apikey" value="${config['sab_apikey']}" size="36">
                                 </div>
                                 <div class="row">
                                     <label title="Name of SABnzbd category to add downloads to.">
                                         SABnzbd Category
                                     </label>
-                                    <input type="text" name="sab_category" value="${config['sab_cat']}" size="20">
+                                    <input type="text" name="sab_category" value="${config['sab_category']}" size="20">
                                 </div>
                             </fieldset>
 
@@ -200,19 +200,19 @@
                                     <label title="NZBGet username. Leave empty if not applicable">
                                         NZBget Username
                                     </label>
-                                    <input type="text" name="nzbget_username" value="${config['nzbget_user']}" size="20">
+                                    <input type="text" name="nzbget_username" value="${config['nzbget_username']}" size="20">
                                 </div>
                                 <div class="row">
                                     <label title="NZBGet password. Leave empty if not applicable">
                                         NZBget Password
                                     </label>
-                                    <input type="password" name="nzbget_password" value="${config['nzbget_pass']}" size="20">
+                                    <input type="password" name="nzbget_password" value="${config['nzbget_password']}" size="20">
                                 </div>
                                 <div class="row">
                                     <label title="Name of NZBget category to add downloads to.">
                                         NZBget Category
                                     </label>
-                                    <input type="text" name="nzbget_category" value="${config['nzbget_cat']}" size="20">
+                                    <input type="text" name="nzbget_category" value="${config['nzbget_category']}" size="20">
                                 </div>
                                 <%
                                     if config['nzbget_priority'] == -100:
@@ -317,10 +317,27 @@
                                     <input type="text" name="torrentblackhole_dir" value="${config['torrentblackhole_dir']}" size="50">
                                     <small>Folder your Download program watches for Torrents</small>
                                 </div>
-                                <div class="row checkbox">
+
-                                    <label>Open Magnet Links</label>
+                                <div class="row">
-                                    <input type="checkbox" name="open_magnet_links" value="1" ${config['open_magnet_links']}>
+                                    <label>Magnet links</label>
-                                    <small>Allow Headphones to open magnet links</small>
+
+                                    <label class="inline" title="Invoke shell command to open magnet URL.">
+                                        <input type="radio" name="magnet_links" id="magnet_links_0" value="0" ${config['magnet_links_0']}>
+                                        Ignore
+                                    </label>
+
+                                    <label class="inline" title="Use external service to convert magnet links into torrents.">
+                                        <input type="radio" name="magnet_links" id="magnet_links_1" value="1" ${config['magnet_links_1']}>
+                                        Open
+                                    </label>
+
+                                    <label class="inline">
+                                        <input type="radio" name="magnet_links" id="magnet_links_2" value="2" ${config['magnet_links_2']}>
+                                        Convert
+                                    </label>
+                                    <div style="clear: both"></div>
+
+                                    <small>Note: opening magnet URL's is not suitable for headless/console/terminal servers.</small>
                                 </div>
                             </fieldset>
                             <fieldset id="transmission_options">
@@ -331,11 +348,11 @@
                                 </div>
                                 <div class="row">
                                     <label>Transmission Username</label>
-                                    <input type="text" name="transmission_username" value="${config['transmission_user']}" size="30">
+                                    <input type="text" name="transmission_username" value="${config['transmission_username']}" size="30">
                                 </div>
                                 <div class="row">
                                     <label>Transmission Password</label>
-                                    <input type="password" name="transmission_password" value="${config['transmission_pass']}" size="30">
+                                    <input type="password" name="transmission_password" value="${config['transmission_password']}" size="30">
                                 </div>
                                 <div class="row">
                                     <small>Note: With Transmission, you can specify a different download directory for downloads sent from Headphones.
@@ -351,11 +368,11 @@
                                 </div>
                                 <div class="row">
                                     <label>uTorrent Username</label>
-                                    <input type="text" name="utorrent_username" value="${config['utorrent_user']}" size="30">
+                                    <input type="text" name="utorrent_username" value="${config['utorrent_username']}" size="30">
                                 </div>
                                 <div class="row">
                                     <label>uTorrent Password</label>
-                                    <input type="password" name="utorrent_password" value="${config['utorrent_pass']}" size="30">
+                                    <input type="password" name="utorrent_password" value="${config['utorrent_password']}" size="30">
                                 </div>
                                 <div class="row">
                                     <label>uTorrent Label</label>
@@ -397,7 +414,8 @@
                             <fieldset>
                                 <legend>Headphones Indexer</legend>
                                 <div class="row checkbox">
-                                    <input id="use_headphones_indexer" type="checkbox" name="use_headphones_indexer" onclick="initConfigCheckbox($(this));" value="1" ${config['use_headphones_indexer']} /><label>Use Headphones Indexer</label>
+                                    <input id="headphones_indexer" type="checkbox" name="headphones_indexer" onclick="initConfigCheckbox($(this));" value="1" ${config['headphones_indexer']} />
+                                    <label>Use Headphones Indexer</label>
                                 </div>
                                 <div class="config">
                                     <div class="row">
@@ -417,7 +435,7 @@
                             <fieldset>
                                 <legend>Custom Newznab Providers</legend>
                                 <div class="row checkbox">
-                                    <input  id="usenewznab" type="checkbox" name="newznab" onclick="initConfigCheckbox($(this));" value="1" ${config['use_newznab']} /><label>Use Newznab</label>
+                                    <input id="use_newznab" type="checkbox" name="use_newznab" onclick="initConfigCheckbox($(this));" value="1" ${config['use_newznab']} /><label>Use Newznab</label>
                                 </div>
                                 <div id="newznab_providers">
                                     <div class="config" id="newznab1">
@@ -428,7 +446,7 @@
                                         </div>
                                         <div class="row">
                                             <label>Newznab API</label>
-                                            <input type="text" name="newznab_apikey" value="${config['newznab_api']}" size="36">
+                                            <input type="text" name="newznab_apikey" value="${config['newznab_apikey']}" size="36">
                                         </div>
                                         <div class="row checkbox">
                                             <input id="newznab_enabled" type="checkbox" name="newznab_enabled" onclick="initConfigCheckbox($(this));" value="1" ${config['newznab_enabled']} /><label>Enabled</label>
@@ -471,7 +489,7 @@
                             <fieldset>
                                 <legend>NZBs.org</legend>
                                 <div class="row checkbox">
-                                    <input id="usenzbsorg" type="checkbox" name="nzbsorg" onclick="initConfigCheckbox($(this));" value="1" ${config['use_nzbsorg']} /><label>Use NZBs.org</label>
+                                    <input id="use_nzbsorg" type="checkbox" name="use_nzbsorg" onclick="initConfigCheckbox($(this));" value="1" ${config['use_nzbsorg']} /><label>Use NZBs.org</label>
                                 </div>
                                 <div class="config">
                                     <div class="row">
@@ -483,7 +501,7 @@
                             <fieldset>
                                 <legend>omgwtfnzbs</legend>
                                 <div class="row checkbox">
-                                    <input id="useomgwtfnzbs" type="checkbox" name="omgwtfnzbs" onclick="initConfigCheckbox($(this));" value="1" ${config['use_omgwtfnzbs']} /><label>Use omgwtfnzbs</label>
+                                    <input id="use_omgwtfnzbs" type="checkbox" name="use_omgwtfnzbs" onclick="initConfigCheckbox($(this));" value="1" ${config['use_omgwtfnzbs']} /><label>Use omgwtfnzbs</label>
                                 </div>
                                 <div class="config">
                                     <div class="row">
@@ -504,7 +522,7 @@
                                 <fieldset>
                                     <legend>The Pirate Bay</legend>
                                     <div class="row checkbox">
-                                        <input id="usepiratebay" type="checkbox" name="use_piratebay" value="1" ${config['use_piratebay']} /><label>Use The Pirate Bay</label>
+                                        <input id="use_piratebay" type="checkbox" name="use_piratebay" value="1" ${config['use_piratebay']} /><label>Use The Pirate Bay</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
@@ -521,7 +539,7 @@
                                 <fieldset>
                                     <legend>Kick Ass Torrents</legend>
                                     <div class="row checkbox">
-                                        <input id="usekat" type="checkbox" name="use_kat" value="1" ${config['use_kat']} /><label>Use Kick Ass Torrents</label>
+                                        <input id="use_kat" type="checkbox" name="use_kat" value="1" ${config['use_kat']} /><label>Use Kick Ass Torrents</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
@@ -538,7 +556,7 @@
                                 <fieldset>
                                     <legend>Waffles.fm</legend>
                                     <div class="row checkbox">
-                                        <input id="usewaffles" type="checkbox" name="waffles" onclick="initConfigCheckbox($(this));" value="1" ${config['use_waffles']} /><label>Use Waffles.fm</label>
+                                        <input id="use_waffles" type="checkbox" name="use_waffles" onclick="initConfigCheckbox($(this));" value="1" ${config['use_waffles']} /><label>Use Waffles.fm</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
@@ -559,7 +577,7 @@
                                 <fieldset>
                                     <legend>rutracker.org</legend>
                                     <div class="row checkbox">
-                                        <input id="userutracker" type="checkbox" name="rutracker" onclick="initConfigCheckbox($(this));" value="1" ${config['use_rutracker']} /><label>Use rutracker.org</label>
+                                        <input id="use_rutracker" type="checkbox" name="use_rutracker" onclick="initConfigCheckbox($(this));" value="1" ${config['use_rutracker']} /><label>Use rutracker.org</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
@@ -580,7 +598,7 @@
                                 <fieldset>
                                     <legend>What.cd</legend>
                                     <div class="row checkbox">
-                                        <input id="usewhatcd" type="checkbox" name="whatcd" onclick="initConfigCheckbox($(this));" value="1" ${config['use_whatcd']} /><label>Use What.cd</label>
+                                        <input id="use_whatcd" type="checkbox" name="use_whatcd" onclick="initConfigCheckbox($(this));" value="1" ${config['use_whatcd']} /><label>Use What.cd</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
@@ -601,7 +619,7 @@
                                 <fieldset>
                                     <legend>Mininova</legend>
                                      <div class="row checkbox">
-                                        <input id="usemininova" type="checkbox" name="use_mininova" value="1" ${config['use_mininova']} /><label>Use Mininova</label>
+                                        <input id="use_mininova" type="checkbox" name="use_mininova" value="1" ${config['use_mininova']} /><label>Use Mininova</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
@@ -650,15 +668,15 @@
                                     <div id="preferred_bitrate_options">
                                         <div class="row">
                                             Target bitrate
-                                            <input type="text" class="override-float" name="preferred_bitrate" value="${config['pref_bitrate']}" size="3">kbps<br>
+                                            <input type="text" class="override-float" name="preferred_bitrate" value="${config['preferred_bitrate']}" size="3">kbps<br>
                                         </div>
                                         <div class="row">
                                             <span style="padding-left: 20px">
-                                                Reject if <strong>less than</strong> <input type="text" class="override-float" name="preferred_bitrate_low_buffer" value="${config['pref_bitrate_low']}" size="3">% or <strong>more than</strong> <input type="text" class="override-float" name="preferred_bitrate_high_buffer" value="${config['pref_bitrate_high']}" size="3">% of the target size (leave blank for no limit)<br>
+                                                Reject if <strong>less than</strong> <input type="text" class="override-float" name="preferred_bitrate_low_buffer" value="${config['preferred_bitrate_low']}" size="3">% or <strong>more than</strong> <input type="text" class="override-float" name="preferred_bitrate_high_buffer" value="${config['preferred_bitrate_high']}" size="3">% of the target size (leave blank for no limit)<br>
                                             </span>
                                         </div>
                                         <div class="row checkbox left" style="padding-left: 20px">
-                                            <input type="checkbox" name="preferred_bitrate_allow_lossless" value="1" ${config['pref_bitrate_allow_lossless']}>
+                                            <input type="checkbox" name="preferred_bitrate_allow_lossless" value="1" ${config['preferred_bitrate_allow_lossless']}>
                                             <label>Allow lossless if no good lossy match found</label>
                                         </div>
                                         <div class="row checkbox left" style="padding-left: 20px">
@@ -691,6 +709,11 @@
                         <td>
                             <fieldset>
                                 <legend>Post-Processing</legend>
+                                <div class="row checkbox left clearfix">
+                                    <label title="Use associated .cue sheet to split single file albums into multiple tracks. Requires shntool with flac or xld cli (OS X) to be installed.">
+                                        Split single file albums into multiple tracks
+                                        <input type="checkbox" name="cue_split" id="cue_split" value="1" ${config['cue_split']} />
+                                    </label>
                                 <div class="row checkbox left clearfix">
                                     <label title="Freeze the database, so new artists won't be added automatically. Use this if Headphones adds artists because due to wrong snatches. This check is skipped when the folder name is appended with release group ID.">
                                         Freeze database for adding new artist
@@ -745,12 +768,12 @@
                                 </div>
                                 <div class="row">
                                     <label>Path to Destination Folder</label>
-                                    <input type="text" name="destination_dir" value="${config['dest_dir']}" size="50">
+                                    <input type="text" name="destination_dir" value="${config['destination_dir']}" size="50">
                                     <small>e.g. /Users/name/Music/iTunes or /Volumes/share/music</small>
                                 </div>
                                 <div class="row">
                                     <label>Path to Lossless Destination folder (optional)</label>
-                                    <input type="text" name="lossless_destination_dir" value="${config['lossless_dest_dir']}" size="50">
+                                    <input type="text" name="lossless_destination_dir" value="${config['lossless_destination_dir']}" size="50">
                                     <small>Set this if you have a separate directory for lossless music</small>
                                 </div>
                             </fieldset>
@@ -1241,7 +1264,7 @@
                             </div>
                             <div class="row">
                                 <label>Path to Encoder</label>
-                                <input type="text" name="encoderfolder" value="${config['encoderfolder']}" size="43">
+                                <input type="text" name="encoder_path" value="${config['encoder_path']}" size="43">
                             </div>
                         </td>
                         <td>
@@ -1278,6 +1301,11 @@
                                 <div class="row left checkbox">
                                     <input type="checkbox" name="autowant_all" value="1" ${config['autowant_all']} /><label>Automatically mark all albums as wanted</label>
                                 </div>
+                                <div class="row left checkbox">
+                                    <input type="checkbox" title="Automatically mark manually added albums from search results as wanted"
+                                           name="autowant_manually_added" value="1" ${config['autowant_manually_added']} />
+                                    <label>Automatically mark manually added albums as wanted</label>
+                                </div>
                                 <div class="row">
                                     <label>Folder Permissions:</label>
                                     <input type="text" name="folder_permissions" value="${config['folder_permissions']}" size="7">
@@ -1298,7 +1326,7 @@
                                     <select name="interface"><h3>
                                     %for interface in config['interface_list']:
                                     <%
-                                        if interface == headphones.INTERFACE:
+                                        if interface == headphones.CONFIG.INTERFACE:
                                             selected = 'selected="selected"'
                                         else:
                                             selected = ''
@@ -1345,9 +1373,9 @@
                                 <div class="row">
                                     <label>Muscbrainz Mirror</label>
                                     <select name="mirror" id="mirror">
-                                    %for mirror in config['mirror_list']:
+                                    %for mirror in config['mirrorlist']:
                                         <%
-                                            if mirror == headphones.MIRROR:
+                                            if mirror == headphones.CONFIG.MIRROR:
                                                 selected = 'selected="selected"'
                                             else:
                                                 selected = ''
@@ -2008,17 +2036,17 @@
                     $( "#tabs" ).tabs();
                 });
                 initActions();
-                initConfigCheckbox("#use_headphones_indexer");
+                initConfigCheckbox("#headphones_indexer");
-                initConfigCheckbox("#usenewznab");
+                initConfigCheckbox("#use_newznab");
-                initConfigCheckbox("#usenzbsorg");
+                initConfigCheckbox("#use_nzbsorg");
-                initConfigCheckbox("#useomgwtfnzbs");
+                initConfigCheckbox("#use_omgwtfnzbs");
-				initConfigCheckbox("#usekat");
+                initConfigCheckbox("#use_kat");
-                initConfigCheckbox("#usepiratebay");
+                initConfigCheckbox("#use_piratebay");
-                initConfigCheckbox("#usemininova");
+                initConfigCheckbox("#use_mininova");
-                initConfigCheckbox("#usewaffles");
+                initConfigCheckbox("#use_waffles");
-                initConfigCheckbox("#userutracker");
+                initConfigCheckbox("#use_rutracker");
-                initConfigCheckbox("#usewhatcd");
+                initConfigCheckbox("#use_whatcd");
-                initConfigCheckbox("#useapi");
+                initConfigCheckbox("#api_enabled");
                 initConfigCheckbox("#enable_https");
 
 

data/interfaces/default/css/style.css

@@ -336,6 +336,10 @@ form .row label {
   padding-top: 7px;
   width: 175px;
 }
+form .row label.inline {
+  margin-right: 5px;
+  width: auto;
+}
 form .row input {
   margin-right: 5px;
 }

data/interfaces/default/css/style.less

@@ -191,6 +191,11 @@ form {
             line-height: normal;
             padding-top: 7px;
             width: 175px;
+
+            &.inline {
+                margin-right: 5px;
+                width: auto;
+            }
         }
         input { margin-right: 5px; }
         input[type=text], input[type=password] {

data/interfaces/default/history.html

@@ -81,8 +81,11 @@
 				"sInfoFiltered":"(filtered from _MAX_ total items)"},
 			"iDisplayLength": 25,
 			"sPaginationType": "full_numbers",
-			"aaSorting": []
+			"aaSorting": [],
-
+			"fnDrawCallback": function (o) {
+                // Jump to top of page
+                $('html,body').scrollTop(0);
+            }
 		});
 		resetFilters("history");
 	}

data/interfaces/default/index.html

@@ -10,7 +10,7 @@
 				<th id="albumart"></th>
 				<th id="name">Artist Name</th>
 				<th id="status">Status</th>
-				<th id="album">Latest Album</th>
+				<th id="album">Latest Release</th>
 				<th id="have">Have</th>
 			</tr>
 		</thead>
@@ -131,6 +131,10 @@
                     },
                 "fnInitComplete": function(oSettings, json)
                 {
+                },
+                "fnDrawCallback": function (o) {
+                    // Jump to top of page
+                    $('html,body').scrollTop(0);
                 }
             });
             $('#artist_table').on("draw.dt", function () {

data/interfaces/default/logs.html

@@ -65,21 +65,20 @@ $(document).ready(function() {
         					"sInfoEmpty":"Showing 0 to 0 of 0 lines",
         					"sInfoFiltered":"(filtered from _MAX_ total lines)"},
                 "fnRowCallback": function (nRow, aData, iDisplayIndex, iDisplayIndexFull) {
-                                if (aData[1] === "ERROR")
+                    if (aData[1] === "ERROR") {
-                                {
                         $('td', nRow).closest('tr').addClass("gradeX");
-                                }
+                    } else if (aData[1] === "WARNING") {
-                                else if (aData[1] === "WARNING")
-                                {
                         $('td', nRow).closest('tr').addClass("gradeW");
-                                }
+                    } else {
-                                else
-                                {
                         $('td', nRow).closest('tr').addClass("gradeZ");
                     }
 
                     return nRow;
                 },
+                "fnDrawCallback": function (o) {
+                    // Jump to top of page
+                    $('html,body').scrollTop(0);
+                },
                 "fnServerData": function ( sSource, aoData, fnCallback ) {
         			/* Add some extra data to the sender */
         			$.getJSON(sSource, aoData, function (json) {

data/interfaces/default/manage.html

@@ -20,7 +20,7 @@
 					</div>
 				</div>
 			<a class="menu_link_edit" href="manageArtists"><i class="fa fa-pencil"></i> Manage Artists</a>
-			%if not headphones.ADD_ARTISTS:
+			%if not headphones.CONFIG.AUTO_ADD_ARTISTS:
 			<a class="menu_link_edit" href="manageNew"><i class="fa fa-pencil"></i> Manage New Artists</a>
 			%endif
 			<a class="menu_link_edit" href="manageUnmatched"><i class="fa fa-pencil"></i> Manage Unmatched</a>
@@ -53,18 +53,18 @@
 					<br/>
 					<div class="row">
 								<label for="">Path to directory</label>
-					%if headphones.MUSIC_DIR:
+					%if headphones.CONFIG.MUSIC_DIR:
-						<input type="text" value="${headphones.MUSIC_DIR}" name="path" size="70" />
+						<input type="text" value="${headphones.CONFIG.MUSIC_DIR}" name="path" size="70" />
 					%else:
 						<input type="text" value="Enter a Music Directory to scan" onfocus="if
 						(this.value==this.defaultValue) this.value='';" name="path" size="70" />
 					%endif
 					</div>
 					<div class="row checkbox"> 
-					<input type="checkbox" name="libraryscan" id="libraryscan" value="1" ${checked(headphones.LIBRARYSCAN)}><label>Automatically scan library</label>
+					<input type="checkbox" name="libraryscan" id="libraryscan" value="1" ${checked(headphones.CONFIG.LIBRARYSCAN)}><label>Automatically scan library</label>
 					</div>
 					<div class="row checkbox"> 
-					<input type="checkbox" name="autoadd" id="autoadd" value="1" ${checked(headphones.ADD_ARTISTS)}><label>Auto-add new artists</label>
+					<input type="checkbox" name="autoadd" id="autoadd" value="1" ${checked(headphones.CONFIG.AUTO_ADD_ARTISTS)}><label>Auto-add new artists</label>
 					</div>
 								
 				</fieldset>
@@ -83,8 +83,8 @@
 			<div class="row">
 				<label for="">Username</label>
 				<%
-					if headphones.LASTFM_USERNAME:
+					if headphones.CONFIG.LASTFM_USERNAME:
-						lastfmvalue = headphones.LASTFM_USERNAME
+						lastfmvalue = headphones.CONFIG.LASTFM_USERNAME
 					else:
 						lastfmvalue = ''
 				%>

data/interfaces/default/managealbums.html

@@ -112,7 +112,6 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-
 		function initThisPage() {
 			$('#album_table').dataTable({
 				"bDestroy": true,
@@ -138,7 +137,11 @@
 					"sInfoFiltered":"(filtered from _MAX_ total albums)",
 					"sSearch": ""},
 				"bPaginate": false,
-			"aaSorting": [[5, 'desc']]
+				"aaSorting": [[5, 'desc']],
+				"fnDrawCallback": function (o) {
+	                // Jump to top of page
+	                $('html,body').scrollTop(0);
+	            }
 
 			});
 			resetFilters("albums");

data/interfaces/default/managemanual.html

@@ -85,10 +85,8 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	$(document).ready(function()
+	$(document).ready(function() {
-	{
+		$('#artist_table').dataTable({
-		$('#artist_table').dataTable(
-			{
 			"bStateSave": true,
 			"bPaginate": true,
 			"oLanguage": {
@@ -100,6 +98,10 @@
 					"sEmptyTable": " ",
                     },
             "sPaginationType": "full_numbers",
+            "fnDrawCallback": function (o) {
+                // Jump to top of page
+                $('html,body').scrollTop(0);
+            }
          });
 
 		initActions();

data/interfaces/default/managenew.html

@@ -6,7 +6,7 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a id="menu_link_scan" onclick="doAjaxCall('musicScan?path=${headphones.MUSIC_DIR}&redirect=manageNew',$(this))" data-success="Music library is getting scanned">Scan Music Library</a>
+			<a id="menu_link_scan" onclick="doAjaxCall('musicScan?path=${headphones.CONFIG.MUSIC_DIR}&redirect=manageNew',$(this))" data-success="Music library is getting scanned">Scan Music Library</a>
 		</div>		
 	</div>	
 	<a href="manage" class="back">&laquo; Back to manage overview</a>
@@ -53,17 +53,15 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	$(document).ready(function()
+	$(document).ready(function() {
-	{
+		$('#artist_table').dataTable({
-		$('#artist_table').dataTable(
-			{
 			"aaSorting": [[1, 'asc']],
 			"bStateSave": false,
 			"bPaginate": false,
 			"oLanguage": {
 				"sSearch" : ""},
-				
 		});
+
 		initActions();
 	});
 	</script>

data/interfaces/default/manageunmatched.html

@@ -118,10 +118,8 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	$(document).ready(function()
+	$(document).ready(function() {
-	{
+		$('#artist_table').dataTable({
-		$('#artist_table').dataTable(
-			{
 			"bStateSave": true,
 			"bPaginate": true,
 			"oLanguage": {
@@ -133,6 +131,10 @@
 					"sEmptyTable": " ",
                     },
             "sPaginationType": "full_numbers",
+            "fnDrawCallback": function (o) {
+                // Jump to top of page
+                $('html,body').scrollTop(0);
+            }
          });
 
 initActions();

data/interfaces/default/searchresults.html

@@ -16,6 +16,7 @@
                     <th id="tracks">Tracks</th>
                     <th id="reldate">Date</th>
                     <th id="scoresmall">Score</th>
+                    <th id="mbrelid" style="display:none;"</th>
                 %else:
                     <th id="artistname">Artist Name</th>
                     <th id="score">Score</th>
@@ -38,7 +39,7 @@
                 %>
 				<tr class="grade${grade}">
 				%if type == 'album':
-					<td id="albumart" style=" text-align: center; vertical-align: middle;"><div id="artistImg"><img title="${result['albumid']}" class="albumArt" height="50" width="50" onerror="this.onerror=null;this.src='${caa_group_url}'"></div></td>
+					<td id="albumart" style=" text-align: center; vertical-align: middle;"><div id="artistImg"><img title="${result['albumid']}" class="albumArt" height="50" width="50" onerror="tryCCA(this, '${caa_group_url}')"></div></td>
                 %else:
 					<td id="albumart"><div id="artistImg"><img title="${result['id']}" class="albumArt" height="50" width="50"></div></td>
 				%endif
@@ -50,6 +51,7 @@
                     <td id="reldate">${result['date']}</td>
                     <td id="score"><a href="${result['albumurl']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
                     <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['albumurl']}"><img src="interfaces/default/images/MusicBrainz_Album_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
+                    <td id="mbrelid" style="display:none;">${result['albumid']}</td>
                 %else:
                     <td id="artistname"><a href="addArtist?artistid=${result['id']}" title="${result['uniquename']}">${result['uniquename']}</a></td>
                     <td id="score"><a href="${result['url']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
@@ -72,7 +74,13 @@
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 
 	<script type="text/javascript">
-
+        function tryCCA(element, url) {
+            element.onerror = function() {
+                element.onerror = null;
+                element.src = "interfaces/default/images/no-cover-art.png";
+            };
+            element.src = url;
+        }
     	function getArt() {
     		$("table#searchresults_table tr td#albumart img").each(function(){
     			var id = $(this).attr('title');
@@ -85,7 +93,6 @@
     		});
     	}
     	function initThisPage() {
-            getArt();
     		$('#searchresults_table').dataTable({
 				"bDestroy": true,
 				"aoColumnDefs": [
@@ -100,11 +107,17 @@
 					"sSearch" : ""},
 				"iDisplayLength": 25,
 				"sPaginationType": "full_numbers",
-				"aaSorting": []
+				"aaSorting": [],
+                "fnDrawCallback": function (o) {
+                    // Jump to top of page
+                    $('html,body').scrollTop(0);
+                }
 			});
             $('#searchresults_table').on("draw.dt", function () {
+                getArt();
                 $("img").unveil();
             });
+            getArt();
 			resetFilters("album");
     	}
     	$(document).ready(function(){
@@ -112,4 +125,18 @@
             initThisPage();
     	});
 	</script>
+
+    <script type="text/javascript">
+        <%!
+            # Abuse JSON module for escaping JavaScript
+            import json
+        %>
+        $(document).ready(function() {
+            // Search parameter
+            $("#searchbar input[name=name]").val(${name | json.dumps});
+
+            // Album or artist
+            $("#searchbar select[name=type]").val(${type | json.dumps});
+        });
+    </script>
 </%def>

headphones/albumart.py

@@ -13,15 +13,18 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import request, db
+from headphones import request, db, logger
+
 
 def getAlbumArt(albumid):
     myDB = db.DBConnection()
-    asin = myDB.action('SELECT AlbumASIN from albums WHERE AlbumID=?', [albumid]).fetchone()[0]
+    asin = myDB.action(
+        'SELECT AlbumASIN from albums WHERE AlbumID=?', [albumid]).fetchone()[0]
 
     if asin:
         return 'http://ec1.images-amazon.com/images/P/%s.01.LZZZZZZZ.jpg' % asin
 
+
 def getCachedArt(albumid):
     from headphones import cache
 

headphones/albumswitcher.py

@@ -14,17 +14,22 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import headphones
-from headphones import db, logger
+from headphones import db, logger, cache
+
 
 def switch(AlbumID, ReleaseID):
-    '''
+    """
     Takes the contents from allalbums & alltracks (based on ReleaseID) and switches them into
     the albums & tracks table.
-    '''
+    """
+    logger.debug('Switching allalbums and alltracks')
     myDB = db.DBConnection()
-    oldalbumdata = myDB.action('SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
+    oldalbumdata = myDB.action(
-    newalbumdata = myDB.action('SELECT * from allalbums WHERE ReleaseID=?', [ReleaseID]).fetchone()
+        'SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
-    newtrackdata = myDB.action('SELECT * from alltracks WHERE ReleaseID=?', [ReleaseID]).fetchall()
+    newalbumdata = myDB.action(
+        'SELECT * from allalbums WHERE ReleaseID=?', [ReleaseID]).fetchone()
+    newtrackdata = myDB.action(
+        'SELECT * from alltracks WHERE ReleaseID=?', [ReleaseID]).fetchall()
     myDB.action('DELETE from tracks WHERE AlbumID=?', [AlbumID])
 
     controlValueDict = {"AlbumID": AlbumID}
@@ -42,6 +47,11 @@ def switch(AlbumID, ReleaseID):
 
     myDB.upsert("albums", newValueDict, controlValueDict)
 
+    # Update cache
+    c = cache.Cache()
+    c.remove_from_cache(AlbumID=AlbumID)
+    c.get_artwork_from_cache(AlbumID=AlbumID)
+
     for track in newtrackdata:
 
         controlValueDict = {"TrackID": track['TrackID'],
@@ -63,16 +73,21 @@ def switch(AlbumID, ReleaseID):
 
         myDB.upsert("tracks", newValueDict, controlValueDict)
 
-    # Mark albums as downloaded if they have at least 80% (by default, configurable) of the album
+    # Mark albums as downloaded if they have at least 80% (by default,
+    # configurable) of the album
     total_track_count = len(newtrackdata)
-    have_track_count = len(myDB.select('SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [AlbumID]))
+    have_track_count = len(myDB.select(
+        'SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [AlbumID]))
 
-    if oldalbumdata['Status'] == 'Skipped' and ((have_track_count/float(total_track_count)) >= (headphones.ALBUM_COMPLETION_PCT/100.0)):
+    if oldalbumdata['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
-        myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', AlbumID])
+        myDB.action(
+            'UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', AlbumID])
 
     # Update have track counts on index
-    totaltracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")', [newalbumdata['ArtistID']]))
+    totaltracks = len(myDB.select(
-    havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [newalbumdata['ArtistID']]))
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")', [newalbumdata['ArtistID']]))
+    havetracks = len(myDB.select(
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [newalbumdata['ArtistID']]))
 
     controlValueDict = {"ArtistID": newalbumdata['ArtistID']}
 

headphones/api.py

@@ -15,10 +15,7 @@
 
 from headphones import db, mb, importer, searcher, cache, postprocessor, versioncheck, logger
 
-from xml.dom.minidom import Document
-
 import headphones
-import copy
 import json
 
 cmd_list = ['getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSimilar', 'getHistory', 'getLogs',
@@ -27,6 +24,7 @@ cmd_list = [ 'getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'g
             'shutdown', 'restart', 'update', 'getArtistArt', 'getAlbumArt', 'getArtistInfo', 'getAlbumInfo',
             'getArtistThumb', 'getAlbumThumb', 'choose_specific_download', 'download_specific_release']
 
+
 class Api(object):
 
     def __init__(self):
@@ -41,16 +39,15 @@ class Api(object):
 
         self.callback = None
 
-
     def checkParams(self, *args, **kwargs):
 
-        if not headphones.API_ENABLED:
+        if not headphones.CONFIG.API_ENABLED:
             self.data = 'API not enabled'
             return
-        if not headphones.API_KEY:
+        if not headphones.CONFIG.API_KEY:
             self.data = 'API key not generated'
             return
-        if len(headphones.API_KEY) != 32:
+        if len(headphones.CONFIG.API_KEY) != 32:
             self.data = 'API key not generated correctly'
             return
 
@@ -58,7 +55,7 @@ class Api(object):
             self.data = 'Missing api key'
             return
 
-        if kwargs['apikey'] != headphones.API_KEY:
+        if kwargs['apikey'] != headphones.CONFIG.API_KEY:
             self.data = 'Incorrect API key'
             return
         else:
@@ -82,9 +79,9 @@ class Api(object):
         if self.data == 'OK':
             logger.info('Recieved API command: %s', self.cmd)
             methodToCall = getattr(self, "_" + self.cmd)
-            result = methodToCall(**self.kwargs)
+            methodToCall(**self.kwargs)
             if 'callback' not in self.kwargs:
-                if type(self.data) == type(''):
+                if isinstance(self.data, basestring):
                     return self.data
                 else:
                     return json.dumps(self.data)
@@ -111,7 +108,8 @@ class Api(object):
 
     def _getIndex(self, **kwargs):
 
-        self.data = self._dic_from_query('SELECT * from artists order by ArtistSortName COLLATE NOCASE')
+        self.data = self._dic_from_query(
+            'SELECT * from artists order by ArtistSortName COLLATE NOCASE')
         return
 
     def _getArtist(self, **kwargs):
@@ -122,11 +120,15 @@ class Api(object):
         else:
             self.id = kwargs['id']
 
-        artist = self._dic_from_query('SELECT * from artists WHERE ArtistID="' + self.id + '"')
+        artist = self._dic_from_query(
-        albums = self._dic_from_query('SELECT * from albums WHERE ArtistID="' + self.id + '" order by ReleaseDate DESC')
+            'SELECT * from artists WHERE ArtistID="' + self.id + '"')
-        description = self._dic_from_query('SELECT * from descriptions WHERE ArtistID="' + self.id + '"')
+        albums = self._dic_from_query(
+            'SELECT * from albums WHERE ArtistID="' + self.id + '" order by ReleaseDate DESC')
+        description = self._dic_from_query(
+            'SELECT * from descriptions WHERE ArtistID="' + self.id + '"')
 
-        self.data = { 'artist': artist, 'albums': albums, 'description' : description }
+        self.data = {
+            'artist': artist, 'albums': albums, 'description': description}
         return
 
     def _getAlbum(self, **kwargs):
@@ -137,23 +139,30 @@ class Api(object):
         else:
             self.id = kwargs['id']
 
-        album = self._dic_from_query('SELECT * from albums WHERE AlbumID="' + self.id + '"')
+        album = self._dic_from_query(
-        tracks = self._dic_from_query('SELECT * from tracks WHERE AlbumID="' + self.id + '"')
+            'SELECT * from albums WHERE AlbumID="' + self.id + '"')
-        description = self._dic_from_query('SELECT * from descriptions WHERE ReleaseGroupID="' + self.id + '"')
+        tracks = self._dic_from_query(
+            'SELECT * from tracks WHERE AlbumID="' + self.id + '"')
+        description = self._dic_from_query(
+            'SELECT * from descriptions WHERE ReleaseGroupID="' + self.id + '"')
 
-        self.data = { 'album' : album, 'tracks' : tracks, 'description' : description }
+        self.data = {
+            'album': album, 'tracks': tracks, 'description': description}
         return
 
     def _getHistory(self, **kwargs):
-        self.data = self._dic_from_query('SELECT * from snatched WHERE status NOT LIKE "Seed%" order by DateAdded DESC')
+        self.data = self._dic_from_query(
+            'SELECT * from snatched WHERE status NOT LIKE "Seed%" order by DateAdded DESC')
         return
 
     def _getUpcoming(self, **kwargs):
-        self.data = self._dic_from_query("SELECT * from albums WHERE ReleaseDate > date('now') order by ReleaseDate DESC")
+        self.data = self._dic_from_query(
+            "SELECT * from albums WHERE ReleaseDate > date('now') order by ReleaseDate DESC")
         return
 
     def _getWanted(self, **kwargs):
-        self.data = self._dic_from_query("SELECT * from albums WHERE Status='Wanted'")
+        self.data = self._dic_from_query(
+            "SELECT * from albums WHERE Status='Wanted'")
         return
 
     def _getSimilar(self, **kwargs):
@@ -194,7 +203,7 @@ class Api(object):
 
         try:
             importer.addArtisttoDB(self.id)
-        except Exception, e:
+        except Exception as e:
             self.data = e
 
         return
@@ -244,7 +253,7 @@ class Api(object):
 
         try:
             importer.addArtisttoDB(self.id)
-        except Exception, e:
+        except Exception as e:
             self.data = e
 
         return
@@ -258,7 +267,7 @@ class Api(object):
 
         try:
             importer.addReleaseById(self.id)
-        except Exception, e:
+        except Exception as e:
             self.data = e
 
         return
@@ -314,7 +323,7 @@ class Api(object):
 
     def _getVersion(self, **kwargs):
         self.data = {
-            'git_path' : headphones.GIT_PATH,
+            'git_path': headphones.CONFIG.GIT_PATH,
             'install_type': headphones.INSTALL_TYPE,
             'current_version': headphones.CURRENT_VERSION,
             'latest_version': headphones.LATEST_VERSION,
@@ -402,7 +411,8 @@ class Api(object):
         else:
             self.id = kwargs['id']
 
-        results = searcher.searchforalbum(self.id, choose_specific_download=True)
+        results = searcher.searchforalbum(
+            self.id, choose_specific_download=True)
 
         results_as_dicts = []
 
@@ -438,10 +448,13 @@ class Api(object):
         for kwarg in expected_kwargs:
             del kwargs[kwarg]
 
-        # Handle situations where the torrent url contains arguments that are parsed
+        # Handle situations where the torrent url contains arguments that are
+        # parsed
         if kwargs:
-            import urllib, urllib2
+            import urllib
-            url = urllib2.quote(url, safe=":?/=&") + '&' + urllib.urlencode(kwargs)
+            import urllib2
+            url = urllib2.quote(
+                url, safe=":?/=&") + '&' + urllib.urlencode(kwargs)
 
         try:
             result = [(title, int(size), url, provider, kind)]
@@ -453,5 +466,6 @@ class Api(object):
 
         if data and bestqual:
             myDB = db.DBConnection()
-          album = myDB.action('SELECT * from albums WHERE AlbumID=?', [id]).fetchone()
+            album = myDB.action(
+                'SELECT * from albums WHERE AlbumID=?', [id]).fetchone()
             searcher.send_to_downloader(data, bestqual, album)

headphones/cache.py

@@ -14,14 +14,13 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import glob
-import urllib
 import headphones
 
 from headphones import db, helpers, logger, lastfm, request
 
 LASTFM_API_KEY = "690e1ed3bc00bc91804cd8f7fe5ed6d4"
 
+
 class Cache(object):
     """
     This class deals with getting, storing and serving up artwork (album
@@ -40,7 +39,7 @@ class Cache(object):
     and for info it is <musicbrainzid>.<date>.txt
     """
 
-    path_to_art_cache = os.path.join(headphones.CACHE_DIR, 'artwork')
+    path_to_art_cache = os.path.join(headphones.CONFIG.CACHE_DIR, 'artwork')
 
     def __init__(self):
         self.id = None
@@ -92,7 +91,6 @@ class Cache(object):
 
         return days_old
 
-
     def _is_current(self, filename=None, date=None):
 
         if filename:
@@ -242,6 +240,36 @@ class Cache(object):
 
         return {'artwork': image_url, 'thumbnail': thumb_url}
 
+    def remove_from_cache(self, ArtistID=None, AlbumID=None):
+        """
+        Pass a musicbrainz id to this function (either ArtistID or AlbumID)
+        """
+
+        if ArtistID:
+            self.id = ArtistID
+            self.id_type = 'artist'
+        else:
+            self.id = AlbumID
+            self.id_type = 'album'
+
+        self.query_type = 'artwork'
+
+        if self._exists('artwork'):
+            for artwork_file in self.artwork_files:
+                try:
+                    os.remove(artwork_file)
+                except:
+                    logger.warn('Error deleting file from the cache: %s', artwork_file)
+
+        self.query_type = 'thumb'
+
+        if self._exists('thumb'):
+            for thumb_file in self.thumb_files:
+                try:
+                    os.remove(thumb_file)
+                except Exception:
+                    logger.warn('Error deleting file from the cache: %s', thumb_file)
+
     def _update_cache(self):
         '''
         Since we call the same url for both info and artwork, we'll update both at the same time
@@ -249,6 +277,7 @@ class Cache(object):
         myDB = db.DBConnection()
 
         # Since lastfm uses release ids rather than release group ids for albums, we have to do a artist + album search for albums
+        # Exception is when adding albums manually, then we should use release id
         if self.id_type == 'artist':
 
             data = lastfm.request_lastfm("artist.getinfo", mbid=self.id, api_key=LASTFM_API_KEY)
@@ -278,8 +307,13 @@ class Cache(object):
 
         else:
 
-            dbartist = myDB.action('SELECT ArtistName, AlbumTitle FROM albums WHERE AlbumID=?', [self.id]).fetchone()
+            dbalbum = myDB.action('SELECT ArtistName, AlbumTitle, ReleaseID FROM albums WHERE AlbumID=?', [self.id]).fetchone()
-            data = lastfm.request_lastfm("album.getinfo", artist=dbartist['ArtistName'], album=dbartist['AlbumTitle'], api_key=LASTFM_API_KEY)
+            if dbalbum['ReleaseID'] != self.id:
+                data = lastfm.request_lastfm("album.getinfo", mbid=dbalbum['ReleaseID'], api_key=LASTFM_API_KEY)
+                if not data:
+                    data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'], album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)
+            else:
+                data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'], album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)
 
             if not data:
                 return
@@ -340,7 +374,7 @@ class Cache(object):
                 if not os.path.isdir(self.path_to_art_cache):
                     try:
                         os.makedirs(self.path_to_art_cache)
-                    except Exception, e:
+                    except Exception as e:
                         logger.error('Unable to create artwork cache dir. Error: %s', e)
                         self.artwork_errors = True
                         self.artwork_url = image_url
@@ -395,6 +429,7 @@ class Cache(object):
                     self.thumb_errors = True
                     self.thumb_url = image_url
 
+
 def getArtwork(ArtistID=None, AlbumID=None):
 
     c = Cache()
@@ -409,6 +444,7 @@ def getArtwork(ArtistID=None, AlbumID=None):
         artwork_file = os.path.basename(artwork_path)
         return "cache/artwork/" + artwork_file
 
+
 def getThumb(ArtistID=None, AlbumID=None):
 
     c = Cache()
@@ -423,6 +459,7 @@ def getThumb(ArtistID=None, AlbumID=None):
         thumbnail_file = os.path.basename(artwork_path)
         return "cache/artwork/" + thumbnail_file
 
+
 def getInfo(ArtistID=None, AlbumID=None):
 
     c = Cache()
@@ -431,6 +468,7 @@ def getInfo(ArtistID=None, AlbumID=None):
 
     return info_dict
 
+
 def getImageLinks(ArtistID=None, AlbumID=None):
 
     c = Cache()

headphones/classes.py

@@ -17,16 +17,16 @@
 ## Stolen from Sick-Beard's classes.py ##
 #########################################
 
-import headphones
 
 import urllib
-import datetime
 
 from common import USER_AGENT
 
+
 class HeadphonesURLopener(urllib.FancyURLopener):
     version = USER_AGENT
 
+
 class AuthURLOpener(HeadphonesURLopener):
     """
     URLOpener class that supports http auth without needing interactive password entry.
@@ -35,6 +35,7 @@ class AuthURLOpener(HeadphonesURLopener):
     user: username to use for HTTP auth
     pw: password to use for HTTP auth
     """
+
     def __init__(self, user, pw):
         self.username = user
         self.password = pw
@@ -65,6 +66,7 @@ class AuthURLOpener(HeadphonesURLopener):
         self.numTries = 0
         return HeadphonesURLopener.open(self, url)
 
+
 class SearchResult:
     """
     Represents a search result from an indexer.
@@ -87,7 +89,7 @@ class SearchResult:
 
     def __str__(self):
 
-        if self.provider == None:
+        if self.provider is None:
             return "Invalid provider, unable to print self"
 
         myString = self.provider.name + " @ " + self.url + "\n"
@@ -96,24 +98,28 @@ class SearchResult:
             myString += "  " + extra + "\n"
         return myString
 
+
 class NZBSearchResult(SearchResult):
     """
     Regular NZB result with an URL to the NZB
     """
     resultType = "nzb"
 
+
 class NZBDataSearchResult(SearchResult):
     """
     NZB result where the actual NZB XML data is stored in the extraInfo
     """
     resultType = "nzbdata"
 
+
 class TorrentSearchResult(SearchResult):
     """
     Torrent result with an URL to the torrent
     """
     resultType = "torrent"
 
+
 class Proper:
     def __init__(self, name, url, date):
         self.name = name

headphones/common.py

@@ -18,7 +18,10 @@ Created on Aug 1, 2011
 
 @author: Michael
 '''
-import platform, operator, os, re
+import platform
+import operator
+import os
+import re
 
 from headphones import version
 
@@ -44,6 +47,7 @@ ARCHIVED = 6 # releases that you don't have locally (counts toward download comp
 IGNORED = 7 # releases that you don't want included in your download stats
 SNATCHED_PROPER = 9 # qualified with quality
 
+
 class Quality:
 
     NONE = 0

headphones/config.py

@@ -0,0 +1,434 @@
+import headphones.logger
+import itertools
+import os
+import re
+from configobj import ConfigObj
+
+
+def bool_int(value):
+    """
+    Casts a config value into a 0 or 1
+    """
+    if isinstance(value, basestring):
+        if value.lower() in ('', '0', 'false', 'f', 'no', 'n', 'off'):
+            value = 0
+    return int(bool(value))
+
+_CONFIG_DEFINITIONS = {
+    'ADD_ALBUM_ART': (int, 'General', 0),
+    'ADVANCEDENCODER': (str, 'General', ''),
+    'ALBUM_ART_FORMAT': (str, 'General', 'folder'),
+    # This is used in importer.py to determine how complete an album needs to
+    # be - to be considered "downloaded". Percentage from 0-100
+    'ALBUM_COMPLETION_PCT': (int, 'Advanced', 80),
+    'API_ENABLED': (int, 'General', 0),
+    'API_KEY': (str, 'General', ''),
+    'AUTOWANT_ALL': (int, 'General', 0),
+    'AUTOWANT_MANUALLY_ADDED': (int, 'General', 1),
+    'AUTOWANT_UPCOMING': (int, 'General', 1),
+    'AUTO_ADD_ARTISTS': (int, 'General', 1),
+    'BITRATE': (int, 'General', 192),
+    'BLACKHOLE': (int, 'General', 0),
+    'BLACKHOLE_DIR': (str, 'General', ''),
+    'BOXCAR_ENABLED': (int, 'Boxcar', 0),
+    'BOXCAR_ONSNATCH': (int, 'Boxcar', 0),
+    'BOXCAR_TOKEN': (str, 'Boxcar', ''),
+    'CACHE_DIR': (str, 'General', ''),
+    'CACHE_SIZEMB': (int, 'Advanced', 32),
+    'CHECK_GITHUB': (int, 'General', 1),
+    'CHECK_GITHUB_INTERVAL': (int, 'General', 360),
+    'CHECK_GITHUB_ON_STARTUP': (int, 'General', 1),
+    'CLEANUP_FILES': (int, 'General', 0),
+    'CONFIG_VERSION': (str, 'General', '0'),
+    'CORRECT_METADATA': (int, 'General', 0),
+    'CUE_SPLIT': (int, 'General', 1),
+    'CUSTOMHOST': (str, 'General', 'localhost'),
+    'CUSTOMPORT': (int, 'General', 5000),
+    'CUSTOMSLEEP': (int, 'General', 1),
+    'DELETE_LOSSLESS_FILES': (int, 'General', 1),
+    'DESTINATION_DIR': (str, 'General', ''),
+    'DETECT_BITRATE': (int, 'General', 0),
+    'DOWNLOAD_DIR': (str, 'General', ''),
+    'DOWNLOAD_SCAN_INTERVAL': (int, 'General', 5),
+    'DOWNLOAD_TORRENT_DIR': (str, 'General', ''),
+    'DO_NOT_OVERRIDE_GIT_BRANCH': (int, 'General', 0),
+    'EMBED_ALBUM_ART': (int, 'General', 0),
+    'EMBED_LYRICS': (int, 'General', 0),
+    'ENABLE_HTTPS': (int, 'General', 0),
+    'ENCODER': (str, 'General', 'ffmpeg'),
+    'ENCODERFOLDER': (str, 'General', ''),
+    'ENCODERLOSSLESS': (int, 'General', 1),
+    'ENCODEROUTPUTFORMAT': (str, 'General', 'mp3'),
+    'ENCODERQUALITY': (int, 'General', 2),
+    'ENCODERVBRCBR': (str, 'General', 'cbr'),
+    'ENCODER_MULTICORE': (int, 'General', 0),
+    'ENCODER_MULTICORE_COUNT': (int, 'General', 0),
+    'ENCODER_PATH': (str, 'General', ''),
+    'EXTRAS': (str, 'General', ''),
+    'EXTRA_NEWZNABS': (list, 'Newznab', ''),
+    'FILE_FORMAT': (str, 'General', 'Track Artist - Album [Year] - Title'),
+    'FILE_PERMISSIONS': (str, 'General', '0644'),
+    'FILE_UNDERSCORES': (int, 'General', 0),
+    'FOLDER_FORMAT': (str, 'General', 'Artist/Album [Year]'),
+    'FOLDER_PERMISSIONS': (str, 'General', '0755'),
+    'FREEZE_DB': (int, 'General', 0),
+    'GIT_BRANCH': (str, 'General', 'master'),
+    'GIT_PATH': (str, 'General', ''),
+    'GIT_USER': (str, 'General', 'rembo10'),
+    'GROWL_ENABLED': (int, 'Growl', 0),
+    'GROWL_HOST': (str, 'Growl', ''),
+    'GROWL_ONSNATCH': (int, 'Growl', 0),
+    'GROWL_PASSWORD': (str, 'Growl', ''),
+    'HEADPHONES_INDEXER': (bool_int, 'General', False),
+    'HPPASS': (str, 'General', ''),
+    'HPUSER': (str, 'General', ''),
+    'HTTPS_CERT': (str, 'General', ''),
+    'HTTPS_KEY': (str, 'General', ''),
+    'HTTP_HOST': (str, 'General', '0.0.0.0'),
+    'HTTP_PASSWORD': (str, 'General', ''),
+    'HTTP_PORT': (int, 'General', 8181),
+    'HTTP_PROXY': (int, 'General', 0),
+    'HTTP_ROOT': (str, 'General', '/'),
+    'HTTP_USERNAME': (str, 'General', ''),
+    'IGNORED_WORDS': (str, 'General', ''),
+    'INCLUDE_EXTRAS': (int, 'General', 0),
+    'INTERFACE': (str, 'General', 'default'),
+    'JOURNAL_MODE': (str, 'Advanced', 'wal'),
+    'KAT': (int, 'Kat', 0),
+    'KAT_PROXY_URL': (str, 'Kat', ''),
+    'KAT_RATIO': (str, 'Kat', ''),
+    'KEEP_NFO': (int, 'General', 0),
+    'KEEP_TORRENT_FILES': (int, 'General', 0),
+    'LASTFM_USERNAME': (str, 'General', ''),
+    'LAUNCH_BROWSER': (int, 'General', 1),
+    'LIBRARYSCAN': (int, 'General', 1),
+    'LIBRARYSCAN_INTERVAL': (int, 'General', 300),
+    'LMS_ENABLED': (int, 'LMS', 0),
+    'LMS_HOST': (str, 'LMS', ''),
+    'LOG_DIR': (str, 'General', ''),
+    'LOSSLESS_BITRATE_FROM': (int, 'General', 0),
+    'LOSSLESS_BITRATE_TO': (int, 'General', 0),
+    'LOSSLESS_DESTINATION_DIR': (str, 'General', ''),
+    'MB_IGNORE_AGE': (int, 'General', 365),
+    'MININOVA': (int, 'Mininova', 0),
+    'MININOVA_RATIO': (str, 'Mininova', ''),
+    'MIRROR': (str, 'General', 'musicbrainz.org'),
+    'MOVE_FILES': (int, 'General', 0),
+    'MPC_ENABLED': (bool_int, 'MPC', False),
+    'MUSIC_DIR': (str, 'General', ''),
+    'MUSIC_ENCODER': (int, 'General', 0),
+    'NEWZNAB': (int, 'Newznab', 0),
+    'NEWZNAB_APIKEY': (str, 'Newznab', ''),
+    'NEWZNAB_ENABLED': (int, 'Newznab', 1),
+    'NEWZNAB_HOST': (str, 'Newznab', ''),
+    'NMA_APIKEY': (str, 'NMA', ''),
+    'NMA_ENABLED': (int, 'NMA', 0),
+    'NMA_ONSNATCH': (int, 'NMA', 0),
+    'NMA_PRIORITY': (int, 'NMA', 0),
+    'NUMBEROFSEEDERS': (str, 'General', '10'),
+    'NZBGET_CATEGORY': (str, 'NZBget', ''),
+    'NZBGET_HOST': (str, 'NZBget', ''),
+    'NZBGET_PASSWORD': (str, 'NZBget', ''),
+    'NZBGET_PRIORITY': (int, 'NZBget', 0),
+    'NZBGET_USERNAME': (str, 'NZBget', 'nzbget'),
+    'NZBSORG': (int, 'NZBsorg', 0),
+    'NZBSORG_HASH': (str, 'NZBsorg', ''),
+    'NZBSORG_UID': (str, 'NZBsorg', ''),
+    'NZB_DOWNLOADER': (int, 'General', 0),
+    'OMGWTFNZBS': (int, 'omgwtfnzbs', 0),
+    'OMGWTFNZBS_APIKEY': (str, 'omgwtfnzbs', ''),
+    'OMGWTFNZBS_UID': (str, 'omgwtfnzbs', ''),
+    'OPEN_MAGNET_LINKS': (int, 'General', 0),  # 0: Ignore, 1: Open, 2: Convert
+    'MAGNET_LINKS': (int, 'General', 0),
+    'OSX_NOTIFY_APP': (str, 'OSX_Notify', '/Applications/Headphones'),
+    'OSX_NOTIFY_ENABLED': (int, 'OSX_Notify', 0),
+    'OSX_NOTIFY_ONSNATCH': (int, 'OSX_Notify', 0),
+    'PIRATEBAY': (int, 'Piratebay', 0),
+    'PIRATEBAY_PROXY_URL': (str, 'Piratebay', ''),
+    'PIRATEBAY_RATIO': (str, 'Piratebay', ''),
+    'PLEX_CLIENT_HOST': (str, 'Plex', ''),
+    'PLEX_ENABLED': (int, 'Plex', 0),
+    'PLEX_NOTIFY': (int, 'Plex', 0),
+    'PLEX_PASSWORD': (str, 'Plex', ''),
+    'PLEX_SERVER_HOST': (str, 'Plex', ''),
+    'PLEX_UPDATE': (int, 'Plex', 0),
+    'PLEX_USERNAME': (str, 'Plex', ''),
+    'PREFERRED_BITRATE': (str, 'General', ''),
+    'PREFERRED_BITRATE_ALLOW_LOSSLESS': (int, 'General', 0),
+    'PREFERRED_BITRATE_HIGH_BUFFER': (int, 'General', 0),
+    'PREFERRED_BITRATE_LOW_BUFFER': (int, 'General', 0),
+    'PREFERRED_QUALITY': (int, 'General', 0),
+    'PREFERRED_WORDS': (str, 'General', ''),
+    'PREFER_TORRENTS': (int, 'General', 0),
+    'PROWL_ENABLED': (int, 'Prowl', 0),
+    'PROWL_KEYS': (str, 'Prowl', ''),
+    'PROWL_ONSNATCH': (int, 'Prowl', 0),
+    'PROWL_PRIORITY': (int, 'Prowl', 0),
+    'PUSHALOT_APIKEY': (str, 'Pushalot', ''),
+    'PUSHALOT_ENABLED': (int, 'Pushalot', 0),
+    'PUSHALOT_ONSNATCH': (int, 'Pushalot', 0),
+    'PUSHBULLET_APIKEY': (str, 'PushBullet', ''),
+    'PUSHBULLET_DEVICEID': (str, 'PushBullet', ''),
+    'PUSHBULLET_ENABLED': (int, 'PushBullet', 0),
+    'PUSHBULLET_ONSNATCH': (int, 'PushBullet', 0),
+    'PUSHOVER_APITOKEN': (str, 'Pushover', ''),
+    'PUSHOVER_ENABLED': (int, 'Pushover', 0),
+    'PUSHOVER_KEYS': (str, 'Pushover', ''),
+    'PUSHOVER_ONSNATCH': (int, 'Pushover', 0),
+    'PUSHOVER_PRIORITY': (int, 'Pushover', 0),
+    'RENAME_FILES': (int, 'General', 0),
+    'REPLACE_EXISTING_FOLDERS': (int, 'General', 0),
+    'REQUIRED_WORDS': (str, 'General', ''),
+    'RUTRACKER': (int, 'Rutracker', 0),
+    'RUTRACKER_PASSWORD': (str, 'Rutracker', ''),
+    'RUTRACKER_RATIO': (str, 'Rutracker', ''),
+    'RUTRACKER_USER': (str, 'Rutracker', ''),
+    'SAB_APIKEY': (str, 'SABnzbd', ''),
+    'SAB_CATEGORY': (str, 'SABnzbd', ''),
+    'SAB_HOST': (str, 'SABnzbd', ''),
+    'SAB_PASSWORD': (str, 'SABnzbd', ''),
+    'SAB_USERNAME': (str, 'SABnzbd', ''),
+    'SAMPLINGFREQUENCY': (int, 'General', 44100),
+    'SEARCH_INTERVAL': (int, 'General', 1440),
+    'SONGKICK_APIKEY': (str, 'Songkick', 'nd1We7dFW2RqxPw8'),
+    'SONGKICK_ENABLED': (int, 'Songkick', 1),
+    'SONGKICK_FILTER_ENABLED': (int, 'Songkick', 0),
+    'SONGKICK_LOCATION': (str, 'Songkick', ''),
+    'SUBSONIC_ENABLED': (int, 'Subsonic', 0),
+    'SUBSONIC_HOST': (str, 'Subsonic', ''),
+    'SUBSONIC_PASSWORD': (str, 'Subsonic', ''),
+    'SUBSONIC_USERNAME': (str, 'Subsonic', ''),
+    'SYNOINDEX_ENABLED': (int, 'Synoindex', 0),
+    'TORRENTBLACKHOLE_DIR': (str, 'General', ''),
+    'TORRENT_DOWNLOADER': (int, 'General', 0),
+    'TORRENT_REMOVAL_INTERVAL': (int, 'General', 720),
+    'TRANSMISSION_HOST': (str, 'Transmission', ''),
+    'TRANSMISSION_PASSWORD': (str, 'Transmission', ''),
+    'TRANSMISSION_USERNAME': (str, 'Transmission', ''),
+    'TWITTER_ENABLED': (int, 'Twitter', 0),
+    'TWITTER_ONSNATCH': (int, 'Twitter', 0),
+    'TWITTER_PASSWORD': (str, 'Twitter', ''),
+    'TWITTER_PREFIX': (str, 'Twitter', 'Headphones'),
+    'TWITTER_USERNAME': (str, 'Twitter', ''),
+    'UPDATE_DB_INTERVAL': (int, 'General', 24),
+    'USENET_RETENTION': (int, 'General', '1500'),
+    'UTORRENT_HOST': (str, 'uTorrent', ''),
+    'UTORRENT_LABEL': (str, 'uTorrent', ''),
+    'UTORRENT_PASSWORD': (str, 'uTorrent', ''),
+    'UTORRENT_USERNAME': (str, 'uTorrent', ''),
+    'VERIFY_SSL_CERT': (bool_int, 'Advanced', 1),
+    'WAFFLES': (int, 'Waffles', 0),
+    'WAFFLES_PASSKEY': (str, 'Waffles', ''),
+    'WAFFLES_RATIO': (str, 'Waffles', ''),
+    'WAFFLES_UID': (str, 'Waffles', ''),
+    'WHATCD': (int, 'What.cd', 0),
+    'WHATCD_PASSWORD': (str, 'What.cd', ''),
+    'WHATCD_RATIO': (str, 'What.cd', ''),
+    'WHATCD_USERNAME': (str, 'What.cd', ''),
+    'XBMC_ENABLED': (int, 'XBMC', 0),
+    'XBMC_HOST': (str, 'XBMC', ''),
+    'XBMC_NOTIFY': (int, 'XBMC', 0),
+    'XBMC_PASSWORD': (str, 'XBMC', ''),
+    'XBMC_UPDATE': (int, 'XBMC', 0),
+    'XBMC_USERNAME': (str, 'XBMC', ''),
+    'XLDPROFILE': (str, 'General', '')
+}
+
+# pylint:disable=R0902
+# it might be nice to refactor for fewer instance variables
+class Config(object):
+    """ Wraps access to particular values in a config file """
+
+    def __init__(self, config_file):
+        """ Initialize the config with values from a file """
+        self._config_file = config_file
+        self._config = ConfigObj(self._config_file, encoding='utf-8')
+        for key in _CONFIG_DEFINITIONS.keys():
+            self.check_setting(key)
+        self.ENCODER_MULTICORE_COUNT = max(0, self.ENCODER_MULTICORE_COUNT)
+        self._upgrade()
+
+    def _define(self, name):
+        key = name.upper()
+        ini_key = name.lower()
+        definition = _CONFIG_DEFINITIONS[key]
+        if len(definition) == 3:
+            definition_type, section, default = definition
+        else:
+            definition_type, section, _, default = definition
+        return key, definition_type, section, ini_key, default
+
+    def check_section(self, section):
+        """ Check if INI section exists, if not create it """
+        if section not in self._config:
+            self._config[section] = {}
+            return True
+        else:
+            return False
+
+    def check_setting(self, key):
+        """ Cast any value in the config to the right type or use the default """
+        key, definition_type, section, ini_key, default = self._define(key)
+        self.check_section(section)
+        try:
+            my_val = definition_type(self._config[section][ini_key])
+        except Exception:
+            my_val = definition_type(default)
+            self._config[section][ini_key] = my_val
+        return my_val
+
+    def write(self):
+        """ Make a copy of the stored config and write it to the configured file """
+        new_config = ConfigObj(encoding="UTF-8")
+        new_config.filename = self._config_file
+
+        # first copy over everything from the old config, even if it is not
+        # correctly defined to keep from losing data
+        for key, subkeys in self._config.items():
+            if key not in new_config:
+                new_config[key] = {}
+            for subkey, value in subkeys.items():
+                new_config[key][subkey] = value
+
+        # next make sure that everything we expect to have defined is so
+        for key in _CONFIG_DEFINITIONS.keys():
+            key, definition_type, section, ini_key, default = self._define(key)
+            self.check_setting(key)
+            if section not in new_config:
+                new_config[section] = {}
+            new_config[section][ini_key] = self._config[section][ini_key]
+
+        # Write it to file
+        headphones.logger.info("Writing configuration to file")
+
+        try:
+            new_config.write()
+        except IOError as e:
+            headphones.logger.error("Error writing configuration file: %s", e)
+
+    def get_extra_newznabs(self):
+        """ Return the extra newznab tuples """
+        extra_newznabs = list(
+            itertools.izip(*[itertools.islice(self.EXTRA_NEWZNABS, i, None, 3)
+            for i in range(3)])
+        )
+        return extra_newznabs
+
+    def clear_extra_newznabs(self):
+        """ Forget about the configured extra newznabs """
+        self.EXTRA_NEWZNABS = []
+
+    def add_extra_newznab(self, newznab):
+        """ Add a new extra newznab """
+        extra_newznabs = self.EXTRA_NEWZNABS
+        for item in newznab:
+            extra_newznabs.append(item)
+        self.EXTRA_NEWZNABS = extra_newznabs
+
+    def __getattr__(self, name):
+        """
+        Returns something from the ini unless it is a real property
+        of the configuration object or is not all caps.
+        """
+        if not re.match(r'[A-Z_]+$', name):
+            return super(Config, self).__getattr__(name)
+        else:
+            return self.check_setting(name)
+
+    def __setattr__(self, name, value):
+        """
+        Maps all-caps properties to ini values unless they exist on the
+        configuration object.
+        """
+        if not re.match(r'[A-Z_]+$', name):
+            super(Config, self).__setattr__(name, value)
+            return value
+        else:
+            key, definition_type, section, ini_key, default = self._define(name)
+            self._config[section][ini_key] = definition_type(value)
+            return self._config[section][ini_key]
+
+    def process_kwargs(self, kwargs):
+        """
+        Given a big bunch of key value pairs, apply them to the ini.
+        """
+        for name, value in kwargs.items():
+            key, definition_type, section, ini_key, default = self._define(name)
+            self._config[section][ini_key] = definition_type(value)
+
+    def _upgrade(self):
+        """ Update folder formats in the config & bump up config version """
+        if self.CONFIG_VERSION == '0':
+            from headphones.helpers import replace_all
+            file_values = {
+                'tracknumber': 'Track',
+                'title': 'Title',
+                'artist': 'Artist',
+                'album': 'Album',
+                'year': 'Year'
+            }
+            folder_values = {
+                'artist': 'Artist',
+                'album': 'Album',
+                'year': 'Year',
+                'releasetype': 'Type',
+                'first': 'First',
+                'lowerfirst': 'first'
+            }
+            self.FILE_FORMAT = replace_all(self.FILE_FORMAT, file_values)
+            self.FOLDER_FORMAT = replace_all(self.FOLDER_FORMAT, folder_values)
+
+            self.CONFIG_VERSION = '1'
+
+        if self.CONFIG_VERSION == '1':
+            from headphones.helpers import replace_all
+            file_values = {
+                'Track': '$Track',
+                'Title': '$Title',
+                'Artist': '$Artist',
+                'Album': '$Album',
+                'Year': '$Year',
+                'track': '$track',
+                'title': '$title',
+                'artist': '$artist',
+                'album': '$album',
+                'year': '$year'
+            }
+            folder_values = {
+                'Artist': '$Artist',
+                'Album': '$Album',
+                'Year': '$Year',
+                'Type': '$Type',
+                'First': '$First',
+                'artist': '$artist',
+                'album': '$album',
+                'year': '$year',
+                'type': '$type',
+                'first': '$first'
+            }
+            self.FILE_FORMAT = replace_all(self.FILE_FORMAT, file_values)
+            self.FOLDER_FORMAT = replace_all(self.FOLDER_FORMAT, folder_values)
+            self.CONFIG_VERSION = '2'
+
+        if self.CONFIG_VERSION == '2':
+            # Update the config to use direct path to the encoder rather than the encoder folder
+            if self.ENCODERFOLDER:
+                self.ENCODER_PATH = os.path.join(self.ENCODERFOLDER, self.ENCODER)
+            self.CONFIG_VERSION = '3'
+
+        if self.CONFIG_VERSION == '3':
+            # Update the BLACKHOLE option to the NZB_DOWNLOADER format
+            if self.BLACKHOLE:
+                self.NZB_DOWNLOADER = 2
+            self.CONFIG_VERSION = '4'
+
+        # Enable Headphones Indexer if they have a VIP account
+        if self.CONFIG_VERSION == '4':
+            if self.HPUSER and self.HPPASS:
+                self.HEADPHONES_INDEXER = True
+            self.CONFIG_VERSION = '5'
+
+        if self.CONFIG_VERSION == '5':
+            if self.OPEN_MAGNET_LINKS:
+                self.MAGNET_LINKS = 2
+            self.CONFIG_VERSION = '5'

headphones/cuesplit.py

@@ -0,0 +1,661 @@
+#  This file is part of Headphones.
+#
+#  Headphones is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  Headphones is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
+
+# Most of this lifted from here: https://github.com/SzieberthAdam/gneposis-cdgrab
+
+import os
+import sys
+import re
+import subprocess
+import copy
+import glob
+
+import headphones
+from headphones import logger
+from mutagen.flac import FLAC
+
+CUE_HEADER = {
+    'genre': '^REM GENRE (.+?)$',
+    'date': '^REM DATE (.+?)$',
+    'discid': '^REM DISCID (.+?)$',
+    'comment': '^REM COMMENT (.+?)$',
+    'catalog': '^CATALOG (.+?)$',
+    'artist': '^PERFORMER (.+?)$',
+    'title': '^TITLE (.+?)$',
+    'file': '^FILE (.+?) (WAVE|FLAC)$',
+    'accurateripid': '^REM ACCURATERIPID (.+?)$'
+}
+
+CUE_TRACK = 'TRACK (\d\d) AUDIO$'
+
+CUE_TRACK_INFO = {
+    'artist': 'PERFORMER (.+?)$',
+    'title': 'TITLE (.+?)$',
+    'isrc': 'ISRC (.+?)$',
+    'index': 'INDEX (\d\d) (.+?)$'
+}
+
+ALBUM_META_FILE_NAME = 'album.dat'
+SPLIT_FILE_NAME = 'split.dat'
+
+ALBUM_META_ALBUM_BY_CUE = ('artist', 'title', 'date', 'genre')
+
+HTOA_LENGTH_TRIGGER = 3
+
+WAVE_FILE_TYPE_BY_EXTENSION = {
+    '.wav': 'Waveform Audio',
+    '.wv': 'WavPack',
+    '.ape': "Monkey's Audio",
+    '.m4a': 'Apple Lossless',
+    '.flac': 'Free Lossless Audio Codec'
+}
+
+# TODO: Only alow flac for now
+#SHNTOOL_COMPATIBLE = ('Waveform Audio', 'WavPack', 'Free Lossless Audio Codec')
+SHNTOOL_COMPATIBLE = ('Free Lossless Audio Codec')
+
+# TODO: Make this better!
+# this module-level variable is bad. :(
+CUE_META = None
+
+
+def check_splitter(command):
+    '''Check xld or shntools installed'''
+    try:
+        env = os.environ.copy()
+        if 'xld' in command:
+            env['PATH'] += os.pathsep + '/Applications'
+        devnull = open(os.devnull)
+        subprocess.Popen([command], stdout=devnull, stderr=devnull, env=env).communicate()
+    except OSError as e:
+        if e.errno == os.errno.ENOENT:
+            return False
+    return True
+
+
+def split_baby(split_file, split_cmd):
+    '''Let's split baby'''
+    logger.info('Splitting %s...', split_file.decode(headphones.SYS_ENCODING, 'replace'))
+    logger.debug(subprocess.list2cmdline(split_cmd))
+
+    # Prevent Windows from opening a terminal window
+    startupinfo = None
+
+    if headphones.SYS_PLATFORM == "win32":
+        startupinfo = subprocess.STARTUPINFO()
+        try:
+            startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
+        except AttributeError:
+            startupinfo.dwFlags |= subprocess._subprocess.STARTF_USESHOWWINDOW
+
+    env = os.environ.copy()
+    if 'xld' in split_cmd:
+        env['PATH'] += os.pathsep + '/Applications'
+
+    process = subprocess.Popen(split_cmd, startupinfo=startupinfo,
+
+    stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
+    stderr=subprocess.PIPE, env=env)
+    stdout, stderr = process.communicate()
+    if process.returncode:
+        logger.error('Split failed for %s', split_file.decode(headphones.SYS_ENCODING, 'replace'))
+        out = stdout if stdout else stderr
+        logger.error('Error details: %s', out.decode(headphones.SYS_ENCODING, 'replace'))
+        return False
+    else:
+        logger.info('Split success %s', split_file.decode(headphones.SYS_ENCODING, 'replace'))
+        return True
+
+
+def check_list(list, ignore=0):
+    '''Checks a list for None elements. If list have None (after ignore index) then it should pass only if all elements
+    are None threreafter. Returns a tuple without the None entries.'''
+
+    if ignore:
+        try:
+            list[int(ignore)]
+        except:
+            raise ValueError('non-integer ignore index or ignore index not in list')
+
+    list1 = list[:ignore]
+    list2 = list[ignore:]
+
+    try:
+        first_none = list2.index(None)
+    except:
+        return tuple(list1 + list2)
+
+    for i in range(first_none, len(list2)):
+        if list2[i]:
+            raise ValueError('non-None entry after None entry in list at index {0}'.format(i))
+
+    while True:
+        list2.remove(None)
+        try:
+            list2.index(None)
+        except:
+            break
+
+    return tuple(list1 + list2)
+
+
+def trim_cue_entry(string):
+    '''Removes leading and trailing "s.'''
+    if string[0] == '"' and string[-1] == '"':
+        string = string[1:-1]
+    return string
+
+
+def int_to_str(value, length=2):
+    '''Converts integer to string eg 3 to "03"'''
+    try:
+        int(value)
+    except:
+        raise ValueError('expected an integer value')
+
+    content = str(value)
+    while len(content) < length:
+        content = '0' + content
+    return content
+
+
+class Directory:
+    def __init__(self, path):
+        self.path = path
+        self.name = os.path.split(self.path)[-1]
+        self.content = []
+        self.update()
+
+    def filter(self, classname):
+        content = []
+        for c in self.content:
+            if c.__class__.__name__ == classname:
+                content.append(c)
+        return content
+
+    def tracks(self, ext=None, split=False):
+        content = []
+        for c in self.content:
+            ext_match = False
+            if c.__class__.__name__ == 'WaveFile':
+                if not ext or (ext and ext == c.name_ext):
+                    ext_match = True
+                if ext_match and c.track_nr:
+                    if not split or (split and c.split_file):
+                        content.append(c)
+        return content
+
+    def update(self):
+        def check_match(filename):
+            for i in self.content:
+                if i.name == filename:
+                    return True
+            return False
+
+        def identify_track_number(filename):
+            if 'split-track' in filename:
+                search = re.search('split-track(\d\d)', filename)
+                if search:
+                    n = int(search.group(1))
+                    if n:
+                        return n
+            for n in range(0, 100):
+                search = re.search(int_to_str(n), filename)
+                if search:
+                    # TODO: not part of other value such as year
+                    return n
+
+        list_dir = glob.glob1(self.path, '*')
+
+        # TODO: for some reason removes only one file
+        rem_list = []
+        for i in self.content:
+            if i.name not in list_dir:
+                rem_list.append(i)
+        for i in rem_list:
+            self.content.remove(i)
+
+        for i in list_dir:
+            if not check_match(i):
+                # music file
+                if os.path.splitext(i)[-1] in WAVE_FILE_TYPE_BY_EXTENSION.keys():
+                    track_nr = identify_track_number(i)
+                    if track_nr:
+                        self.content.append(WaveFile(self.path + os.sep + i, track_nr=track_nr))
+                    else:
+                        self.content.append(WaveFile(self.path + os.sep + i))
+
+                # cue file
+                elif os.path.splitext(i)[-1] == '.cue':
+                    self.content.append(CueFile(self.path + os.sep + i))
+
+                # meta file
+                elif i == ALBUM_META_FILE_NAME:
+                    self.content.append(MetaFile(self.path + os.sep + i))
+
+                # directory
+                elif os.path.isdir(i):
+                    self.content.append(Directory(self.path + os.sep + i))
+
+                else:
+                    self.content.append(File(self.path + os.sep + i))
+
+
+class File(object):
+    def __init__(self, path):
+        self.path = path
+        self.name = os.path.split(self.path)[-1]
+
+        self.name_name = ''.join(os.path.splitext(self.name)[:-1])
+        self.name_ext = os.path.splitext(self.name)[-1]
+        self.split_file = True if self.name_name[:11] == 'split-track' else False
+
+    def get_name(self, ext=True, cmd=False):
+
+        if ext is True:
+            content = self.name
+        elif ext is False:
+            content = self.name_name
+        elif ext[0] == '.':
+            content = self.name_name + ext
+        else:
+            raise ValueError('ext parameter error')
+
+        if cmd:
+            content = content.replace(' ', '\ ')
+
+        return content
+
+
+class CueFile(File):
+    def __init__(self, path):
+
+        def header_parser():
+            global line_content
+            c = self.content.splitlines()
+            header_dict = {}
+            #remaining_headers = CUE_HEADER
+            remaining_headers = copy.copy(CUE_HEADER)
+            line_index = 0
+            match = True
+            while match:
+                match = False
+                saved_match = None
+                line_content = c[line_index]
+                for e in remaining_headers:
+                    search_result = re.search(remaining_headers[e], line_content, re.I)
+                    if search_result:
+                        search_content = trim_cue_entry(search_result.group(1))
+                        header_dict[e] = search_content
+                        saved_match = e
+                        match = True
+                        line_index += 1
+                if saved_match:
+                    del remaining_headers[saved_match]
+            return header_dict, line_index
+
+        def track_parser(start_line):
+            c = self.content.splitlines()
+            line_index = start_line
+            line_content = c[line_index]
+            search_result = re.search(CUE_TRACK, line_content, re.I)
+            if not search_result:
+                raise ValueError('inconsistent CUE sheet, TRACK expected at line {0}'.format(line_index + 1))
+            track_nr = int(search_result.group(1))
+            line_index += 1
+            next_track = False
+            track_meta = {}
+            # we make room for future indexes
+            track_meta['index'] = [None for m in range(100)]
+
+            while not next_track:
+                if line_index < len(c):
+                    line_content = c[line_index]
+
+                    artist_search = re.search(CUE_TRACK_INFO['artist'], line_content, re.I)
+                    title_search = re.search(CUE_TRACK_INFO['title'], line_content, re.I)
+                    isrc_search = re.search(CUE_TRACK_INFO['isrc'], line_content, re.I)
+                    index_search = re.search(CUE_TRACK_INFO['index'], line_content, re.I)
+
+                    if artist_search:
+                        if trim_cue_entry(artist_search.group(1)) != self.header['artist']:
+                            track_meta['artist'] = trim_cue_entry(artist_search.group(1))
+                        line_index += 1
+                    elif title_search:
+                        track_meta['title'] = trim_cue_entry(title_search.group(1))
+                        line_index += 1
+                    elif isrc_search:
+                        track_meta['isrc'] = trim_cue_entry(isrc_search.group(1))
+                        line_index += 1
+                    elif index_search:
+                        track_meta['index'][int(index_search.group(1))] = index_search.group(2)
+                        line_index += 1
+                    elif re.search(CUE_TRACK, line_content, re.I):
+                        next_track = True
+                    elif line_index == len(c) - 1 and not line_content:
+                        # last line is empty
+                        line_index += 1
+                    elif re.search('FLAGS DCP$', line_content, re.I):
+                        track_meta['dcpflag'] = True
+                        line_index += 1
+                    else:
+                        raise ValueError('unknown entry in track error, line {0}'.format(line_index + 1))
+                else:
+                    next_track = True
+
+            track_meta['index'] = check_list(track_meta['index'], ignore=1)
+
+            return track_nr, track_meta, line_index
+
+        super(CueFile, self).__init__(path)
+
+        try:
+            with open(self.name) as cue_file:
+                self.content = cue_file.read()
+        except:
+            self.content = None
+
+        if not self.content:
+            try:
+                 with open(self.name, encoding="cp1252") as cue_file:
+                     self.content = cue_file.read()
+            except:
+                raise ValueError('Cant encode CUE Sheet.')
+
+        if self.content[0] == u'\ufeff':
+            self.content = self.content[1:]
+
+        header = header_parser()
+
+        self.header = header[0]
+
+        line_index = header[1]
+
+        # we make room for tracks
+        tracks = [None for m in range(100)]
+
+        while line_index < len(self.content.splitlines()):
+            parsed_track = track_parser(line_index)
+            line_index = parsed_track[2]
+            tracks[parsed_track[0]] = parsed_track[1]
+
+        self.tracks = check_list(tracks, ignore=1)
+
+    def get_meta(self):
+        content = ''
+        for i in ALBUM_META_ALBUM_BY_CUE:
+            if self.header.get(i):
+                content += i + '\t' + self.header[i] + '\n'
+            else:
+                content += i + '\t' + '\n'
+
+        for i in range(len(self.tracks)):
+            if self.tracks[i]:
+                if self.tracks[i].get('artist'):
+                    content += 'track' + int_to_str(i) + 'artist' + '\t' + self.tracks[i].get('artist') + '\n'
+                if self.tracks[i].get('title'):
+                    content += 'track' + int_to_str(i) + 'title' + '\t' + self.tracks[i].get('title') + '\n'
+        return content
+
+    def htoa(self):
+        '''Returns true if Hidden Track exists.'''
+        if int(self.tracks[1]['index'][1][-5:-3]) >= HTOA_LENGTH_TRIGGER:
+            return True
+        return False
+
+    def breakpoints(self):
+        '''Returns track break points. Identical as CUETools' cuebreakpoints, with the exception of my standards for HTOA.'''
+        content = ''
+        for t in range(len(self.tracks)):
+            if t == 1 and not self.htoa():
+                content += ''
+            elif t >= 1:
+                t_index = self.tracks[t]['index']
+                content += t_index[1]
+                if (t < len(self.tracks) - 1):
+                    content += '\n'
+        return content
+
+
+class MetaFile(File):
+    def __init__(self, path):
+        super(MetaFile, self).__init__(path)
+        with open(self.path) as meta_file:
+            self.rawcontent = meta_file.read()
+
+        content = {}
+        content['tracks'] = [None for m in range(100)]
+
+        for l in self.rawcontent.splitlines():
+            parsed_line = re.search('^(.+?)\t(.+?)$', l)
+            if parsed_line:
+                if parsed_line.group(1)[:5] == 'track':
+                    parsed_track = re.search('^track(\d\d)(.+?)$', parsed_line.group(1))
+                    if not parsed_track:
+                        raise ValueError('Syntax error in album meta file')
+                    if not content['tracks'][int(parsed_track.group(1))]:
+                        content['tracks'][int(parsed_track.group(1))] = dict()
+                    content['tracks'][int(parsed_track.group(1))][parsed_track.group(2)] = parsed_line.group(2)
+                else:
+                    content[parsed_line.group(1)] = parsed_line.group(2)
+
+        content['tracks'] = check_list(content['tracks'], ignore=1)
+
+        self.content = content
+
+    def flac_tags(self, track_nr):
+        common_tags = dict()
+        freeform_tags = dict()
+
+        # common flac tags
+        common_tags['artist'] = self.content['artist']
+        common_tags['album'] = self.content['title']
+        common_tags['title'] = self.content['tracks'][track_nr]['title']
+        common_tags['tracknumber'] = str(track_nr)
+        common_tags['tracktotal'] = str(len(self.content['tracks']) - 1)
+        if 'date' in self.content:
+            common_tags['date'] = self.content['date']
+        if 'genre' in CUE_META.content:
+            common_tags['genre'] = CUE_META.content['genre']
+
+        #freeform tags
+        #freeform_tags['country'] = self.content['country']
+        #freeform_tags['releasedate'] = self.content['releasedate']
+
+        return common_tags, freeform_tags
+
+    def folders(self):
+        artist = self.content['artist']
+        album = self.content['date'] + ' - ' + self.content['title'] + ' (' + self.content['label'] + ' - ' + self.content['catalog'] + ')'
+        return artist, album
+
+    def complete(self):
+        '''Check MetaFile for containing all data'''
+        self.__init__(self.path)
+        for l in self.rawcontent.splitlines():
+            if re.search('^[0-9A-Za-z]+?\t$', l):
+                return False
+        return True
+
+    def count_tracks(self):
+        '''Returns tracks count'''
+        return len(self.content['tracks']) - self.content['tracks'].count(None)
+
+
+class WaveFile(File):
+    def __init__(self, path, track_nr=None):
+        super(WaveFile, self).__init__(path)
+
+        self.track_nr = track_nr
+        self.type = WAVE_FILE_TYPE_BY_EXTENSION[self.name_ext]
+
+    def filename(self, ext=None, cmd=False):
+        title = CUE_META.content['tracks'][self.track_nr]['title']
+
+        if ext:
+            if ext[0] != '.':
+                ext = '.' + ext
+        else:
+            ext = self.name_ext
+
+        f_name = int_to_str(self.track_nr) + ' - ' + title + ext
+
+        if cmd:
+            f_name = f_name.replace(' ', '\ ')
+
+        f_name = f_name.replace('!', '')
+        f_name = f_name.replace('?', '')
+        f_name = f_name.replace('/', ';')
+
+        return f_name
+
+    def tag(self):
+        if self.type == 'Free Lossless Audio Codec':
+            f = FLAC(self.name)
+            tags = CUE_META.flac_tags(self.track_nr)
+            for t in tags[0]:
+                f[t] = tags[0][t]
+            f.save()
+
+    def mutagen(self):
+        if self.type == 'Free Lossless Audio Codec':
+            return FLAC(self.name)
+
+def split(albumpath):
+    global CUE_META
+    os.chdir(albumpath)
+    base_dir = Directory(os.getcwd())
+
+    cue = None
+    wave = None
+
+    # determining correct cue file
+    # if perfect match found
+    for _cue in base_dir.filter('CueFile'):
+        for _wave in base_dir.filter('WaveFile'):
+            if _cue.header['file'] == _wave.name:
+                logger.info('CUE Sheet found: {0}'.format(_cue.name))
+                logger.info('Music file found: {0}'.format(_wave.name))
+                cue = _cue
+                wave = _wave
+    # if no perfect match found then try without extensions
+    if not cue and not wave:
+        logger.info('No match for music files, trying to match without extensions...')
+        for _cue in base_dir.filter('CueFile'):
+            for _wave in base_dir.filter('WaveFile'):
+                if ''.join(os.path.splitext(_cue.header['file'])[:-1]) == _wave.name_name:
+                    logger.info('Possible CUE Sheet found: {0}'.format(_cue.name))
+                    logger.info('CUE Sheet refers music file: {0}'.format(_cue.header['file']))
+                    logger.info('Possible Music file found: {0}'.format(_wave.name))
+                    cue = _cue
+                    wave = _wave
+                    cue.header['file'] = wave.name
+    # if still no match then raise an exception
+    if not cue and not wave:
+        raise ValueError('No music file match found!')
+
+    # Split with xld or shntool
+    splitter = 'shntool'
+    xldprofile = None
+
+    # use xld profile to split cue
+    if headphones.CONFIG.ENCODER == 'xld' and headphones.CONFIG.MUSIC_ENCODER and headphones.CONFIG.XLDPROFILE:
+        import getXldProfile
+        xldprofile, xldformat, _ = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
+        if not xldformat:
+            raise ValueError('Details for xld profile "%s" not found, cannot split cue' % (xldprofile))
+        else:
+            if headphones.CONFIG.ENCODERFOLDER:
+                splitter = os.path.join(headphones.CONFIG.ENCODERFOLDER, 'xld')
+            else:
+                splitter = 'xld'
+    # use standard xld command to split cue
+    elif sys.platform == 'darwin':
+        splitter = 'xld'
+        if not check_splitter(splitter):
+            splitter = 'shntool'
+
+    if splitter == 'shntool' and not check_splitter(splitter):
+            raise ValueError('Command not found, ensure shntools with FLAC or xld (OS X) installed')
+
+    # Determine if file can be split (only flac allowed for shntools)
+    if 'xld' in splitter and wave.name_ext not in WAVE_FILE_TYPE_BY_EXTENSION.keys() or \
+            wave.type not in SHNTOOL_COMPATIBLE:
+        raise ValueError('Cannot split, audio file has unsupported extension')
+
+    # Split with xld
+    if 'xld' in splitter:
+        cmd = [splitter]
+        cmd.extend([wave.name])
+        cmd.extend(['-c'])
+        cmd.extend([cue.name])
+        if xldprofile:
+            cmd.extend(['--profile'])
+            cmd.extend([xldprofile])
+        else:
+            cmd.extend(['-f'])
+            cmd.extend(['flac'])
+        cmd.extend(['-o'])
+        cmd.extend([base_dir.path])
+        split = split_baby(wave.name, cmd)
+    else:
+
+        # Split with shntool
+
+        # generate temporary metafile describing the cue
+        with open(ALBUM_META_FILE_NAME, mode='w') as meta_file:
+            meta_file.write(cue.get_meta())
+        base_dir.content.append(MetaFile(os.path.abspath(ALBUM_META_FILE_NAME)))
+        # check metafile for completeness
+        if not base_dir.filter('MetaFile'):
+            raise ValueError('Cue Meta file {0} missing!'.format(ALBUM_META_FILE_NAME))
+        else:
+            CUE_META = base_dir.filter('MetaFile')[0]
+
+        with open(SPLIT_FILE_NAME, mode='w') as split_file:
+            split_file.write(cue.breakpoints())
+
+        cmd = ['shntool']
+        cmd.extend(['split'])
+        cmd.extend(['-f'])
+        cmd.extend([SPLIT_FILE_NAME])
+        cmd.extend(['-o'])
+        cmd.extend(['flac'])
+        cmd.extend([wave.name])
+        split = split_baby(wave.name, cmd)
+        os.remove(SPLIT_FILE_NAME)
+        base_dir.update()
+
+        # tag FLAC files
+        if split and CUE_META.count_tracks() == len(base_dir.tracks(ext='.flac', split=True)):
+            for t in base_dir.tracks(ext='.flac', split=True):
+                logger.info('Tagging {0}...'.format(t.name))
+                t.tag()
+
+        # rename FLAC files
+        if split and CUE_META.count_tracks() == len(base_dir.tracks(ext='.flac', split=True)):
+            for t in base_dir.tracks(ext='.flac', split=True):
+                if t.name != t.filename():
+                    logger.info('Renaming {0} to {1}...'.format(t.name, t.filename()))
+                    os.rename(t.name, t.filename())
+
+        os.remove(ALBUM_META_FILE_NAME)
+
+    if not split:
+        raise ValueError('Failed to split, check logs')
+    else:
+        # Rename original file
+        os.rename(wave.name, wave.name + '.original')
+        return True

headphones/db.py

@@ -21,23 +21,24 @@ from __future__ import with_statement
 
 import os
 import sqlite3
-import threading
-import time
 
 import headphones
 
 from headphones import logger
 
+
 def dbFilename(filename="headphones.db"):
 
     return os.path.join(headphones.DATA_DIR, filename)
 
+
 def getCacheSize():
     #this will protect against typecasting problems produced by empty string and None settings
-    if not headphones.CACHE_SIZEMB:
+    if not headphones.CONFIG.CACHE_SIZEMB:
         #sqlite will work with this (very slowly)
         return 0
-    return int(headphones.CACHE_SIZEMB)
+    return int(headphones.CONFIG.CACHE_SIZEMB)
+
 
 class DBConnection:
 
@@ -48,21 +49,21 @@ class DBConnection:
         #don't wait for the disk to finish writing
         self.connection.execute("PRAGMA synchronous = OFF")
         #journal disabled since we never do rollbacks
-        self.connection.execute("PRAGMA journal_mode = %s" % headphones.JOURNAL_MODE)
+        self.connection.execute("PRAGMA journal_mode = %s" % headphones.CONFIG.JOURNAL_MODE)
         #64mb of cache memory,probably need to make it user configurable
         self.connection.execute("PRAGMA cache_size=-%s" % (getCacheSize() * 1024))
         self.connection.row_factory = sqlite3.Row
 
     def action(self, query, args=None):
 
-        if query == None:
+        if query is None:
             return
 
         sqlResult = None
 
         try:
             with self.connection as c:
-                if args == None:
+                if args is None:
                     sqlResult = c.execute(query)
                 else:
                     sqlResult = c.execute(query, args)
@@ -84,7 +85,7 @@ class DBConnection:
 
         sqlResults = self.action(query, args).fetchall()
 
-        if sqlResults == None or sqlResults == [None]:
+        if sqlResults is None or sqlResults == [None]:
             return []
 
         return sqlResults

headphones/exceptions.py

@@ -13,11 +13,13 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
+
 class HeadphonesException(Exception):
     """
     Generic Headphones Exception - should never be thrown, only subclassed
     """
 
+
 class NewzbinAPIThrottled(HeadphonesException):
     """
     Newzbin has throttled us, deal with it

headphones/getXldProfile.py

@@ -2,9 +2,9 @@ import os.path
 import plistlib
 import sys
 import xml.parsers.expat as expat
-import commands
 from headphones import logger
 
+
 def getXldProfile(xldProfile):
     xldProfileNotFound = xldProfile
     expandedPath = os.path.expanduser('~/Library/Preferences/jp.tmkk.XLD.plist')
@@ -16,12 +16,12 @@ def getXldProfile(xldProfile):
             preferences = plistlib.Plist.fromFile(expandedPath)
         except (ImportError):
             os.system("/usr/bin/plutil -convert binary1 %s" % expandedPath)
-            logger.info('The plist at "%s" has a date in it, and therefore is not useable.' % expandedPath)
+            logger.info('The plist at "%s" has a date in it, and therefore is not useable.', expandedPath)
             return(xldProfileNotFound, None, None)
     except (ImportError):
-        logger.info('The plist at "%s" has a date in it, and therefore is not useable.' % expandedPath)
+        logger.info('The plist at "%s" has a date in it, and therefore is not useable.', expandedPath)
     except:
-        logger.info('Unexpected error:', sys.exc_info()[0])
+        logger.info('Unexpected error: %s', sys.exc_info()[0])
         return(xldProfileNotFound, None, None)
 
     xldProfile = xldProfile.lower()

headphones/helpers.py

@@ -31,6 +31,7 @@ RE_FEATURING = re.compile(r"[fF]t\.|[fF]eaturing|[fF]eat\.|\b[wW]ith\b|&|vs\.")
 RE_CD_ALBUM = re.compile(r"\(?((CD|disc)\s*[0-9]+)\)?", re.I)
 RE_CD = re.compile(r"^(CD|dics)\s*[0-9]+$", re.I)
 
+
 def multikeysort(items, columns):
     comparers = [((itemgetter(col[1:].strip()), -1) if col.startswith('-') else (itemgetter(col.strip()), 1)) for col in columns]
 
@@ -44,12 +45,14 @@ def multikeysort(items, columns):
 
     return sorted(items, cmp=comparer)
 
+
 def checked(variable):
     if variable:
         return 'Checked'
     else:
         return ''
 
+
 def radio(variable, pos):
 
     if variable == pos:
@@ -57,6 +60,7 @@ def radio(variable, pos):
     else:
         return ''
 
+
 def latinToAscii(unicrap):
     """
     From couch potato
@@ -90,7 +94,7 @@ def latinToAscii(unicrap):
 
     r = ''
     for i in unicrap:
-        if xlate.has_key(ord(i)):
+        if ord(i) in xlate:
             r += xlate[ord(i)]
         elif ord(i) >= 0x80:
             pass
@@ -98,6 +102,7 @@ def latinToAscii(unicrap):
             r += str(i)
     return r
 
+
 def convert_milliseconds(ms):
 
     seconds = ms / 1000
@@ -109,6 +114,7 @@ def convert_milliseconds(ms):
 
     return minutes
 
+
 def convert_seconds(s):
 
     gmtime = time.gmtime(s)
@@ -119,15 +125,18 @@ def convert_seconds(s):
 
     return minutes
 
+
 def today():
     today = datetime.date.today()
     yyyymmdd = datetime.date.isoformat(today)
     return yyyymmdd
 
+
 def now():
     now = datetime.datetime.now()
     return now.strftime("%Y-%m-%d %H:%M:%S")
 
+
 def get_age(date):
 
     try:
@@ -142,17 +151,20 @@ def get_age(date):
 
     return days_old
 
+
 def bytes_to_mb(bytes):
 
     mb = int(bytes) / 1048576
     size = '%.1f MB' % mb
     return size
 
+
 def mb_to_bytes(mb_str):
     result = re.search('^(\d+(?:\.\d+)?)\s?(?:mb)?', mb_str, flags=re.I)
     if result:
         return int(float(result.group(1)) * 1048576)
 
+
 def piratesize(size):
     split = size.split(" ")
     factor = float(split[0])
@@ -170,6 +182,7 @@ def piratesize(size):
 
     return size
 
+
 def replace_all(text, dic, normalize=False):
 
     if not text:
@@ -177,13 +190,17 @@ def replace_all(text, dic, normalize=False):
 
     for i, j in dic.iteritems():
         if normalize:
+            try:
                 if sys.platform == 'darwin':
                     j = unicodedata.normalize('NFD', j)
                 else:
                     j = unicodedata.normalize('NFC', j)
+            except TypeError:
+                j = unicodedata.normalize('NFC', j.decode(headphones.SYS_ENCODING, 'replace'))
         text = text.replace(i, j)
     return text
 
+
 def replace_illegal_chars(string, type="file"):
     if type == "file":
         string = re.sub('[\?"*:|<>/]', '_', string)
@@ -192,6 +209,7 @@ def replace_illegal_chars(string, type="file"):
 
     return string
 
+
 def cleanName(string):
 
     pass1 = latinToAscii(string).lower()
@@ -199,6 +217,7 @@ def cleanName(string):
 
     return out_string
 
+
 def cleanTitle(title):
 
     title = re.sub('[\.\-\/\_]', ' ', title).lower()
@@ -210,6 +229,7 @@ def cleanTitle(title):
 
     return title
 
+
 def split_path(f):
     """
     Split a path into components, starting with the drive letter (if any). Given
@@ -219,7 +239,7 @@ def split_path(f):
     components = []
     drive, path = os.path.splitdrive(f)
 
-    # Stip the folder from the path, iterate until nothing is left
+    # Strip the folder from the path, iterate until nothing is left
     while True:
         path, folder = os.path.split(path)
 
@@ -241,6 +261,7 @@ def split_path(f):
     # Done
     return components
 
+
 def expand_subfolders(f):
     """
     Try to expand a given folder and search for subfolders containing media
@@ -307,23 +328,15 @@ def expand_subfolders(f):
     logger.debug("Expanded subfolders in folder: %s", media_folders)
     return media_folders
 
+
 def extract_data(s):
 
     s = s.replace('_', ' ')
 
     #headphones default format
-    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\[(?P<year>.*?)\]', re.VERBOSE)
+    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s[\[\(](?P<year>.*?)[\]\)]', re.VERBOSE)
     match = pattern.match(s)
 
-    if match:
-        name = match.group("name")
-        album = match.group("album")
-        year = match.group("year")
-        return (name, album, year)
-
-    #newzbin default format
-    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\((?P<year>\d+?\))', re.VERBOSE)
-    match = pattern.match(s)
     if match:
         name = match.group("name")
         album = match.group("album")
@@ -343,6 +356,7 @@ def extract_data(s):
     else:
         return (None, None, None)
 
+
 def extract_metadata(f):
     """
     Scan all files in the given directory and decide on an artist, album and
@@ -422,7 +436,7 @@ def extract_metadata(f):
 
     # (Lots of) different artists. Could be a featuring album, so test for this.
     if len(artists) > 1 and len(albums) == 1:
-        split_artists = [ RE_FEATURING.split(artist) for artist in artists ]
+        split_artists = [RE_FEATURING.split(x) for x in artists]
         featurings = [len(split_artist) - 1 for split_artist in split_artists]
         logger.info("Album seem to feature %d different artists", sum(featurings))
 
@@ -441,6 +455,79 @@ def extract_metadata(f):
 
     return (None, None, None)
 
+
+def get_downloaded_track_list(albumpath):
+    """
+     Return a list of audio files for the given directory.
+     """
+    downloaded_track_list = []
+
+    for root, dirs, files in os.walk(albumpath):
+        for _file in files:
+            extension = os.path.splitext(_file)[1].lower()[1:]
+            if extension in headphones.MEDIA_FORMATS:
+                downloaded_track_list.append(os.path.join(root, _file))
+
+    return downloaded_track_list
+
+
+def preserve_torrent_direcory(albumpath):
+    """
+     Copy torrent directory to headphones-modified to keep files for seeding.
+     """
+    from headphones import logger
+    new_folder = os.path.join(albumpath, 'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
+    logger.info("Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")
+    try:
+        shutil.copytree(albumpath, new_folder)
+        return new_folder
+    except Exception as e:
+        logger.warn("Cannot copy/move files to temp folder: " + \
+                    new_folder.decode(headphones.SYS_ENCODING, 'replace') + \
+                    ". Not continuing. Error: " + str(e))
+        return None
+
+
+def cue_split(albumpath):
+    """
+     Attempts to check and split audio files by a cue for the given directory.
+     """
+    # Walk directory and scan all media files
+    count = 0
+    cue_count = 0
+    cue_dirs = []
+
+    for root, dirs, files in os.walk(albumpath):
+        for _file in files:
+            extension = os.path.splitext(_file)[1].lower()[1:]
+            if extension in headphones.MEDIA_FORMATS:
+                count += 1
+            elif extension == 'cue':
+                cue_count += 1
+                if root not in cue_dirs:
+                    cue_dirs.append(root)
+
+    # Split cue
+    if cue_count and cue_count >= count and cue_dirs:
+
+        from headphones import logger, cuesplit
+        logger.info("Attempting to split audio files by cue")
+
+        cwd = os.getcwd()
+        for cue_dir in cue_dirs:
+            try:
+                cuesplit.split(cue_dir)
+            except Exception as e:
+                os.chdir(cwd)
+                logger.warn("Cue not split: " + str(e))
+                return False
+
+        os.chdir(cwd)
+        return True
+
+    return False
+
+
 def extract_logline(s):
     # Default log format
     pattern = re.compile(r'(?P<timestamp>.*?)\s\-\s(?P<level>.*?)\s*\:\:\s(?P<thread>.*?)\s\:\s(?P<message>.*)', re.VERBOSE)
@@ -454,14 +541,11 @@ def extract_logline(s):
     else:
         return None
 
+
 def extract_song_data(s):
+    from headphones import logger
 
     #headphones default format
-    music_dir = headphones.MUSIC_DIR
-    folder_format = headphones.FOLDER_FORMAT
-    file_format = headphones.FILE_FORMAT
-
-    full_format = os.path.join(headphones.MUSIC_DIR)
     pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\[(?P<year>.*?)\]', re.VERBOSE)
     match = pattern.match(s)
 
@@ -485,6 +569,7 @@ def extract_song_data(s):
         logger.info("Couldn't parse %s into a valid Newbin format", s)
         return (name, album, year)
 
+
 def smartMove(src, dest, delete=True):
 
     from headphones import logger
@@ -506,7 +591,7 @@ def smartMove(src, dest, delete=True):
                 try:
                     os.rename(src, os.path.join(source_dir, newfile))
                     filename = newfile
-                except Exception, e:
+                except Exception as e:
                     logger.warn('Error renaming %s: %s', src.decode(headphones.SYS_ENCODING, 'replace'), e)
                 break
 
@@ -516,7 +601,7 @@ def smartMove(src, dest, delete=True):
         else:
             shutil.copy(os.path.join(source_dir, filename), os.path.join(dest, filename))
             return True
-    except Exception, e:
+    except Exception as e:
         logger.warn('Error moving file %s: %s', filename.decode(headphones.SYS_ENCODING, 'replace'), e)
 
 #########################
@@ -525,11 +610,15 @@ def smartMove(src, dest, delete=True):
 
 # TODO: Grab config values from sab to know when these options are checked. For now we'll just iterate through all combinations
 
+
 def sab_replace_dots(name):
     return name.replace('.', ' ')
+
+
 def sab_replace_spaces(name):
     return name.replace(' ', '_')
 
+
 def sab_sanitize_foldername(name):
     """ Return foldername with dodgy chars converted to safe ones
         Remove any leading and trailing dot and space characters
@@ -571,12 +660,14 @@ def sab_sanitize_foldername(name):
 
     return name
 
+
 def split_string(mystring, splitvar=','):
     mylist = []
     for each_word in mystring.split(splitvar):
         mylist.append(each_word.strip())
     return mylist
 
+
 def create_https_certificates(ssl_cert, ssl_key):
     """
     Stolen from SickBeard (http://github.com/midgetspy/Sick-Beard):
@@ -586,7 +677,7 @@ def create_https_certificates(ssl_cert, ssl_key):
 
     try:
         from OpenSSL import crypto
-        from lib.certgen import createKeyPair, createCertRequest, createCertificate, TYPE_RSA, serial
+        from certgen import createKeyPair, createCertRequest, createCertificate, TYPE_RSA, serial
     except:
         logger.warn("pyOpenSSL module missing, please install to enable HTTPS")
         return False

headphones/importer.py

@@ -17,9 +17,7 @@ from headphones import logger, helpers, db, mb, lastfm
 
 from beets.mediafile import MediaFile
 
-import os
 import time
-import threading
 import headphones
 
 blacklisted_special_artist_names = ['[anonymous]', '[data]', '[no artist]',
@@ -32,6 +30,7 @@ blacklisted_special_artists = ['f731ccc4-e22a-43af-a747-64213329e088',
     '125ec42a-7229-4250-afc5-e057484327fe',
     '89ad4ac3-39f7-470e-963a-56509c546377']
 
+
 def is_exists(artistid):
     myDB = db.DBConnection()
 
@@ -52,7 +51,6 @@ def artistlist_to_mbids(artistlist, forced=False):
         if not artist and not (artist == ' '):
             continue
 
-
         # If adding artists through Manage New Artists, they're coming through as non-unicode (utf-8?)
         # and screwing everything up
         if not isinstance(artist, unicode):
@@ -105,12 +103,14 @@ def artistlist_to_mbids(artistlist, forced=False):
     except Exception as e:
         logger.warn('Failed to update arist information from Last.fm: %s' % e)
 
+
 def addArtistIDListToDB(artistidlist):
     # Used to add a list of artist IDs to the database in a single thread
     logger.debug("Importer: Adding artist ids %s" % artistidlist)
     for artistid in artistidlist:
         addArtisttoDB(artistid)
 
+
 def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
     # Putting this here to get around the circular import. We're using this to update thumbnails for artist/albums
@@ -140,8 +140,8 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
     if not dbartist:
         newValueDict = {"ArtistName": "Artist ID: %s" % (artistid),
                         "Status": "Loading",
-                        "IncludeExtras": headphones.INCLUDE_EXTRAS,
+                        "IncludeExtras": headphones.CONFIG.INCLUDE_EXTRAS,
-                        "Extras":        headphones.EXTRAS }
+                        "Extras": headphones.CONFIG.EXTRAS}
     else:
         newValueDict = {"Status": "Loading"}
 
@@ -172,7 +172,6 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
     else:
         sortname = artist['artist_name']
 
-
     logger.info(u"Now adding/updating: " + artist['artist_name'])
     controlValueDict = {"ArtistID": artistid}
     newValueDict = {"ArtistName": artist['artist_name'],
@@ -227,7 +226,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
         rgid = rg['id']
         skip_log = 0
         #Make a user configurable variable to skip update of albums with release dates older than this date (in days)
-        pause_delta = headphones.MB_IGNORE_AGE
+        pause_delta = headphones.CONFIG.MB_IGNORE_AGE
 
         rg_exists = myDB.action("SELECT * from albums WHERE AlbumID=?", [rg['id']]).fetchone()
 
@@ -240,7 +239,6 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                 check_release_date = None
                 new_release_group = True
 
-
             if new_release_group:
                 logger.info("[%s] Now adding: %s (New Release Group)" % (artist['artist_name'], rg['title']))
                 new_releases = mb.get_new_releases(rgid, includeExtras)
@@ -414,13 +412,13 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
                 newValueDict['DateAdded'] = today
 
-                if headphones.AUTOWANT_ALL:
+                if headphones.CONFIG.AUTOWANT_ALL:
                     newValueDict['Status'] = "Wanted"
-                elif album['ReleaseDate'] > today and headphones.AUTOWANT_UPCOMING:
+                elif album['ReleaseDate'] > today and headphones.CONFIG.AUTOWANT_UPCOMING:
                     newValueDict['Status'] = "Wanted"
                 # Sometimes "new" albums are added to musicbrainz after their release date, so let's try to catch these
                 # The first test just makes sure we have year-month-day
-                elif helpers.get_age(album['ReleaseDate']) and helpers.get_age(today) - helpers.get_age(album['ReleaseDate']) < 21 and headphones.AUTOWANT_UPCOMING:
+                elif helpers.get_age(album['ReleaseDate']) and helpers.get_age(today) - helpers.get_age(album['ReleaseDate']) < 21 and headphones.CONFIG.AUTOWANT_UPCOMING:
                     newValueDict['Status'] = "Wanted"
                 else:
                     newValueDict['Status'] = "Skipped"
@@ -464,11 +462,11 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
             marked_as_downloaded = False
 
             if rg_exists:
-                if rg_exists['Status'] == 'Skipped' and ((have_track_count/float(total_track_count)) >= (headphones.ALBUM_COMPLETION_PCT/100.0)):
+                if rg_exists['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
                     myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
                     marked_as_downloaded = True
             else:
-                if ((have_track_count/float(total_track_count)) >= (headphones.ALBUM_COMPLETION_PCT/100.0)):
+                if ((have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
                     myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
                     marked_as_downloaded = True
 
@@ -478,7 +476,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
             # Start a search for the album if it's new, hasn't been marked as
             # downloaded and autowant_all is selected. This search is deferred,
             # in case the search failes and the rest of the import will halt.
-            if not rg_exists and not marked_as_downloaded and headphones.AUTOWANT_ALL:
+            if not rg_exists and not marked_as_downloaded and headphones.CONFIG.AUTOWANT_ALL:
                 album_searches.append(rg['id'])
         else:
             if skip_log == 0:
@@ -504,6 +502,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
         for album_search in album_searches:
             searcher.searchforalbum(albumid=album_search)
 
+
 def finalize_update(artistid, artistname, errors=False):
     # Moving this little bit to it's own function so we can update have tracks & latest album when deleting extras
 
@@ -533,6 +532,7 @@ def finalize_update(artistid, artistname, errors=False):
 
     myDB.upsert("artists", newValueDict, controlValueDict)
 
+
 def addReleaseById(rid, rgid=None):
 
     myDB = db.DBConnection()
@@ -596,9 +596,9 @@ def addReleaseById(rid, rgid=None):
                         "DateAdded": helpers.today(),
                         "Status": "Paused"}
 
-        if headphones.INCLUDE_EXTRAS:
+        if headphones.CONFIG.INCLUDE_EXTRAS:
             newValueDict['IncludeExtras'] = 1
-            newValueDict['Extras'] = headphones.EXTRAS
+            newValueDict['Extras'] = headphones.CONFIG.EXTRAS
 
         myDB.upsert("artists", newValueDict, controlValueDict)
 
@@ -618,7 +618,7 @@ def addReleaseById(rid, rgid=None):
         newValueDict = {"ArtistID": release_dict['artist_id'],
                         "ReleaseID": rgid,
                         "ArtistName": release_dict['artist_name'],
-                        "AlbumTitle":       release_dict['rg_title'],
+                        "AlbumTitle": release_dict['title'] if 'title' in release_dict else release_dict['rg_title'],
                         "AlbumASIN": release_dict['asin'],
                         "ReleaseDate": release_dict['date'],
                         "DateAdded": helpers.today(),
@@ -647,10 +647,10 @@ def addReleaseById(rid, rgid=None):
                         "CleanName": cleanname
                         }
 
-            match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?', [cleanname]).fetchone()
+            match = myDB.action('SELECT Location, BitRate, Format, Matched from have WHERE CleanName=?', [cleanname]).fetchone()
 
             if not match:
-                match = myDB.action('SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [release_dict['artist_name'], release_dict['rg_title'], track['title']]).fetchone()
+                match = myDB.action('SELECT Location, BitRate, Format, Matched from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [release_dict['artist_name'], release_dict['rg_title'], track['title']]).fetchone()
 
             #if not match:
                 #match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
@@ -661,17 +661,26 @@ def addReleaseById(rid, rgid=None):
                 newValueDict['Format'] = match['Format']
                 #myDB.action('DELETE from have WHERE Location=?', [match['Location']])
 
+                # If the album has been scanned before adding the release it will be unmatched, update to matched
+                if match['Matched'] == 'Failed':
+                    myDB.action('UPDATE have SET Matched=? WHERE Location=?', (release_dict['rgid'], match['Location']))
+
             myDB.upsert("tracks", newValueDict, controlValueDict)
 
         # Reset status
         if status == 'Loading':
             controlValueDict = {"AlbumID": rgid}
+            if headphones.CONFIG.AUTOWANT_MANUALLY_ADDED:
                 newValueDict = {"Status": "Wanted"}
+            else:
+                newValueDict = {"Status": "Skipped"}
             myDB.upsert("albums", newValueDict, controlValueDict)
 
         # Start a search for the album
+        if headphones.CONFIG.AUTOWANT_MANUALLY_ADDED:
             import searcher
             searcher.searchforalbum(rgid, False)
+
     elif not rg_exists and not release_dict:
         logger.error("ReleaseGroup does not exist in the database and did not get a valid response from MB. Skipping release.")
         if status == 'Loading':
@@ -680,6 +689,7 @@ def addReleaseById(rid, rgid=None):
     else:
         logger.info('Release ' + str(rid) + " already exists in the database!")
 
+
 def updateFormat():
     myDB = db.DBConnection()
     tracks = myDB.select('SELECT * from tracks WHERE Location IS NOT NULL and Format IS NULL')
@@ -688,7 +698,7 @@ def updateFormat():
         for track in tracks:
             try:
                 f = MediaFile(track['Location'])
-            except Exception, e:
+            except Exception as e:
                 logger.info("Exception from MediaFile for: " + track['Location'] + " : " + str(e))
                 continue
             controlValueDict = {"TrackID": track['TrackID']}
@@ -701,7 +711,7 @@ def updateFormat():
         for track in havetracks:
             try:
                 f = MediaFile(track['Location'])
-            except Exception, e:
+            except Exception as e:
                 logger.info("Exception from MediaFile for: " + track['Location'] + " : " + str(e))
                 continue
             controlValueDict = {"TrackID": track['TrackID']}
@@ -709,6 +719,7 @@ def updateFormat():
             myDB.upsert("have", newValueDict, controlValueDict)
         logger.info('Finished finding media format for %s files' % len(havetracks))
 
+
 def getHybridRelease(fullreleaselist):
     """
     Returns a dictionary of best group of tracks from the list of releases and
@@ -767,8 +778,8 @@ def getHybridRelease(fullreleaselist):
         # Change this value to change the sorting behaviour of none, returning
         # 'None' will put it at the top which was normal behaviour for pre-ngs
         # versions
-        if releaseDate == None:
+        if releaseDate is None:
-            return 'None';
+            return 'None'
 
         if releaseDate.count('-') == 2:
             return releaseDate

headphones/lastfm.py

@@ -13,18 +13,24 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import random
 import time
+import random
+import threading
 import headphones
 
 from headphones import db, logger, request
 
 from collections import defaultdict
 
-TIMEOUT = 60 # seconds
+TIMEOUT = 60.0 # seconds
+REQUEST_LIMIT = 1.0 / 5 # seconds
 ENTRY_POINT = "http://ws.audioscrobbler.com/2.0/"
 API_KEY = "395e6ec6bb557382fc41fde867bce66f"
 
+# Required for API request limit
+lock = threading.Lock()
+
+
 def request_lastfm(method, **kwargs):
     """
     Call a Last.FM API method. Automatically sets the method and API key. Method
@@ -43,7 +49,8 @@ def request_lastfm(method, **kwargs):
     logger.debug("Calling Last.FM method: %s", method)
     logger.debug("Last.FM call parameters: %s", kwargs)
 
-    data = request.request_json(ENTRY_POINT, timeout=TIMEOUT, params=kwargs)
+    data = request.request_json(ENTRY_POINT, timeout=TIMEOUT, params=kwargs,
+        rate_limit=(lock, REQUEST_LIMIT))
 
     # Parse response and check for errors.
     if not data:
@@ -56,6 +63,7 @@ def request_lastfm(method, **kwargs):
 
     return data
 
+
 def getSimilar():
     myDB = db.DBConnection()
     results = myDB.select("SELECT ArtistID from artists ORDER BY HaveTracks DESC")
@@ -101,16 +109,17 @@ def getSimilar():
 
     logger.debug("Inserted %d artists into Last.FM tag cloud", len(top_list))
 
+
 def getArtists():
     myDB = db.DBConnection()
     results = myDB.select("SELECT ArtistID from artists")
 
-    if not headphones.LASTFM_USERNAME:
+    if not headphones.CONFIG.LASTFM_USERNAME:
         logger.warn("Last.FM username not set, not importing artists.")
         return
 
-    logger.info("Fetching artists from Last.FM for username: %s", headphones.LASTFM_USERNAME)
+    logger.info("Fetching artists from Last.FM for username: %s", headphones.CONFIG.LASTFM_USERNAME)
-    data = request_lastfm("library.getartists", limit=10000, user=headphones.LASTFM_USERNAME)
+    data = request_lastfm("library.getartists", limit=10000, user=headphones.CONFIG.LASTFM_USERNAME)
 
     if data and "artists" in data:
         artistlist = []
@@ -130,6 +139,7 @@ def getArtists():
 
         logger.info("Imported %d new artists from Last.FM", len(artistlist))
 
+
 def getTagTopArtists(tag, limit=50):
     myDB = db.DBConnection()
     results = myDB.select("SELECT ArtistID from artists")

headphones/librarysync.py

@@ -14,7 +14,6 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import glob
 import headphones
 
 from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
@@ -22,17 +21,18 @@ from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
 from headphones import db, logger, helpers, importer, lastfm
 
 # You can scan a single directory and append it to the current library by specifying append=True, ArtistID & ArtistName
+
+
 def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=False):
 
-
+    if cron and not headphones.CONFIG.LIBRARYSCAN:
-    if cron and not headphones.LIBRARYSCAN:
         return
 
     if not dir:
-        if not headphones.MUSIC_DIR:
+        if not headphones.CONFIG.MUSIC_DIR:
             return
         else:
-            dir = headphones.MUSIC_DIR
+            dir = headphones.CONFIG.MUSIC_DIR
 
     # If we're appending a dir, it's coming from the post processor which is
     # already bytestring
@@ -78,9 +78,11 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
 
     latest_subdirectory = []
 
-    for r,d,f in os.walk(dir):
+    for r, d, f in os.walk(dir, followlinks=True):
-        #need to abuse slicing to get a copy of the list, doing it directly will skip the element after a deleted one
+        # Need to abuse slicing to get a copy of the list, doing it directly
-        #using a list comprehension will not work correctly for nested subdirectories (os.walk keeps its original list)
+        # will skip the element after a deleted one using a list comprehension
+        # will not work correctly for nested subdirectories (os.walk keeps its
+        # original list)
         for directory in d[:]:
             if directory.startswith("."):
                 d.remove(directory)
@@ -105,7 +107,10 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
                 try:
                     f = MediaFile(song)
                 except (FileTypeError, UnreadableFileError):
-                    logger.error("Cannot read file media file '%s'. It may be corrupted or not a media file.", unicode_song_path)
+                    logger.warning("Cannot read media file '%s', skipping. It may be corrupted or not a media file.", unicode_song_path)
+                    continue
+                except IOError:
+                    logger.warning("Cannnot read media file '%s', skipping. Does the file exists?", unicode_song_path)
                     continue
 
                 # Grab the bitrates for the auto detect bit rate option
@@ -175,7 +180,6 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
 
                 file_count += 1
 
-
     # Now we start track matching
     logger.info("%s new/modified songs found and added to the database" % new_song_count)
     song_list = myDB.action("SELECT * FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"])
@@ -200,7 +204,6 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
         elif latest_artist[song_count] != latest_artist[song_count - 1] and song_count != 0:
             logger.info("Now matching songs by %s" % song['ArtistName'])
 
-        #print song['ArtistName']+' - '+song['AlbumTitle']+' - '+song['TrackTitle']
         song_count += 1
         completion_percentage = float(song_count) / total_number_of_songs * 100
 
@@ -289,7 +292,6 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
 
     logger.info('Completed matching tracks from directory: %s' % dir.decode(headphones.SYS_ENCODING, 'replace'))
 
-
     if not append:
         logger.info('Updating scanned artist track counts')
 
@@ -298,15 +300,30 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
         current_artists = myDB.select('SELECT ArtistName, ArtistID from artists')
 
         #There was a bug where artists with special characters (-,') would show up in new artists.
-        artist_list = [f for f in unique_artists if helpers.cleanName(f).lower() not in [helpers.cleanName(x[0]).lower() for x in current_artists]]
+        artist_list = [
-        artists_checked = [f for f in unique_artists if helpers.cleanName(f).lower() in [helpers.cleanName(x[0]).lower() for x in current_artists]]
+            x for x in unique_artists
+            if helpers.cleanName(x).lower() not in [
+                helpers.cleanName(y[0]).lower()
+                for y in current_artists
+            ]
+        ]
+        artists_checked = [
+            x for x in unique_artists
+            if helpers.cleanName(x).lower() in [
+                helpers.cleanName(y[0]).lower()
+                for y in current_artists
+            ]
+        ]
 
         # Update track counts
 
         for artist in artists_checked:
             # Have tracks are selected from tracks table and not all tracks because of duplicates
             # We update the track count upon an album switch to compliment this
-            havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL', [artist])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artist]))
+            havetracks = (
+                len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL', [artist]))
+                + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artist]))
+            )
             #Note, some people complain about having "artist have tracks" > # of tracks total in artist official releases
             # (can fix by getting rid of second len statement)
             myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistName=?', [havetracks, artist])
@@ -314,7 +331,7 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
         logger.info('Found %i new artists' % len(artist_list))
 
         if len(artist_list):
-            if headphones.ADD_ARTISTS:
+            if headphones.CONFIG.AUTO_ADD_ARTISTS:
                 logger.info('Importing %i new artists' % len(artist_list))
                 importer.artistlist_to_mbids(artist_list)
             else:
@@ -323,8 +340,8 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
                 for artist in artist_list:
                     myDB.action('INSERT OR IGNORE INTO newartists VALUES (?)', [artist])
 
-        if headphones.DETECT_BITRATE:
+        if headphones.CONFIG.DETECT_BITRATE:
-            headphones.PREFERRED_BITRATE = sum(bitrates)/len(bitrates)/1000
+            headphones.CONFIG.PREFERRED_BITRATE = sum(bitrates) / len(bitrates) / 1000
 
     else:
         # If we're appending a new album to the database, update the artists total track counts
@@ -339,6 +356,8 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
     logger.info('Library scan complete')
 
 #ADDED THIS SECTION TO MARK ALBUMS AS DOWNLOADED IF ARTISTS ARE ADDED EN MASSE BEFORE LIBRARY IS SCANNED
+
+
 def update_album_status(AlbumID=None):
     myDB = db.DBConnection()
     logger.info('Counting matched tracks to mark albums as skipped/downloaded')
@@ -360,7 +379,7 @@ def update_album_status(AlbumID=None):
             album_completion = 0
             logger.info('Album %s does not have any tracks in database' % album['AlbumTitle'])
 
-        if album_completion >= headphones.ALBUM_COMPLETION_PCT and album['Status'] == 'Skipped':
+        if album_completion >= headphones.CONFIG.ALBUM_COMPLETION_PCT and album['Status'] == 'Skipped':
             new_album_status = "Downloaded"
 
         # I don't think we want to change Downloaded->Skipped.....

headphones/logger.py

@@ -24,6 +24,7 @@ import headphones
 import threading
 import traceback
 import logging
+import errno
 import sys
 import os
 
@@ -35,8 +36,9 @@ MAX_FILES = 5
 # Headphones logger
 logger = logging.getLogger("headphones")
 
-# Global queue of multiprocessing logging
+# Global queue for multiprocessing logging
-queue = multiprocessing.Queue()
+queue = None
+
 
 class LogListHandler(logging.Handler):
     """
@@ -49,6 +51,7 @@ class LogListHandler(logging.Handler):
 
         headphones.LOG_LIST.insert(0, (helpers.now(), message, record.levelname, record.threadName))
 
+
 @contextlib.contextmanager
 def listener():
     """
@@ -56,6 +59,26 @@ def listener():
     To be used in a with statement in the main process, for multiprocessing.
     """
 
+    global queue
+
+    # Initialize queue if not already done
+    if queue is None:
+        try:
+            queue = multiprocessing.Queue()
+        except OSError as e:
+            queue = False
+
+            # Some machines don't have access to /dev/shm. See
+            # http://stackoverflow.com/questions/2009278 for more information.
+            if e.errno == errno.EACCES:
+                logger.warning("Multiprocess logging disabled, because "
+                    "current user cannot map shared memory. You won't see any" \
+                    "logging generated by the worker processed.")
+
+    # Multiprocess logging may be disabled.
+    if not queue:
+        yield
+    else:
         queue_listener = QueueListener(queue, *logger.handlers)
 
         try:
@@ -64,6 +87,7 @@ def listener():
         finally:
             queue_listener.stop()
 
+
 def initMultiprocessing():
     """
     Remove all handlers and add QueueHandler on top. This should only be called
@@ -71,6 +95,11 @@ def initMultiprocessing():
     completely.
     """
 
+    # Multiprocess logging may be disabled.
+    if not queue:
+        return
+
+    # Remove all handlers and add the Queue handler as the only one.
     for handler in logger.handlers[:]:
         logger.removeHandler(handler)
 
@@ -82,6 +111,7 @@ def initMultiprocessing():
     # Change current thread name for log record
     threading.current_thread().name = multiprocessing.current_process().name
 
+
 def initLogger(console=False, verbose=False):
     """
     Setup logging for Headphones. It uses the logger instance with the name
@@ -110,7 +140,7 @@ def initLogger(console=False, verbose=False):
     logger.setLevel(logging.DEBUG if verbose else logging.INFO)
 
     # Setup file logger
-    filename = os.path.join(headphones.LOG_DIR, FILENAME)
+    filename = os.path.join(headphones.CONFIG.LOG_DIR, FILENAME)
 
     file_formatter = logging.Formatter('%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
     file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE, backupCount=MAX_FILES)
@@ -137,6 +167,7 @@ def initLogger(console=False, verbose=False):
     # Install exception hooks
     initHooks()
 
+
 def initHooks(global_exceptions=True, thread_exceptions=True, pass_original=True):
     """
     This method installs exception catching mechanisms. Any exception caught

headphones/lyrics.py

@@ -18,6 +18,7 @@ import htmlentitydefs
 
 from headphones import logger, request
 
+
 def getLyrics(artist, song):
 
     params = {"artist": artist.encode('utf-8'),
@@ -60,6 +61,7 @@ def getLyrics(artist, song):
 
     return lyrics
 
+
 def convert_html_entities(s):
     matches = re.findall("&#\d+;", s)
     if len(matches) > 0:
@@ -79,7 +81,7 @@ def convert_html_entities(s):
         hits.remove(amp)
     for hit in hits:
         name = hit[1:-1]
-        if htmlentitydefs.name2codepoint.has_key(name):
+        if name in htmlentitydefs.name2codepoint:
             s = s.replace(hit, unichr(htmlentitydefs.name2codepoint[name]))
     s = s.replace(amp, "&")
     return s

headphones/mb.py

@@ -15,7 +15,6 @@
 
 
 from headphones import logger, db, helpers
-from headphones.helpers import multikeysort, replace_all
 
 import time
 import threading
@@ -23,33 +22,38 @@ import headphones
 import musicbrainzngs
 
 try:
+    # pylint:disable=E0611
+    # ignore this error because we are catching the ImportError
     from collections import OrderedDict
+    # pylint:enable=E0611
 except ImportError:
     # Python 2.6.x fallback, from libs
-    import OrderedDict
+    from ordereddict import OrderedDict
 
 mb_lock = threading.Lock()
 
 # Quick fix to add mirror switching on the fly. Need to probably return the mbhost & mbport that's
 # being used, so we can send those values to the log
+
+
 def startmb():
 
     mbuser = None
     mbpass = None
 
-    if headphones.MIRROR == "musicbrainz.org":
+    if headphones.CONFIG.MIRROR == "musicbrainz.org":
         mbhost = "musicbrainz.org"
         mbport = 80
         sleepytime = 1
-    elif headphones.MIRROR == "custom":
+    elif headphones.CONFIG.MIRROR == "custom":
-        mbhost = headphones.CUSTOMHOST
+        mbhost = headphones.CONFIG.CUSTOMHOST
-        mbport = int(headphones.CUSTOMPORT)
+        mbport = int(headphones.CONFIG.CUSTOMPORT)
-        sleepytime = int(headphones.CUSTOMSLEEP)
+        sleepytime = int(headphones.CONFIG.CUSTOMSLEEP)
-    elif headphones.MIRROR == "headphones":
+    elif headphones.CONFIG.MIRROR == "headphones":
         mbhost = "144.76.94.239"
         mbport = 8181
-        mbuser = headphones.HPUSER
+        mbuser = headphones.CONFIG.HPUSER
-        mbpass = headphones.HPPASS
+        mbpass = headphones.CONFIG.HPPASS
         sleepytime = 0
     else:
         return False
@@ -63,7 +67,7 @@ def startmb():
         musicbrainzngs.set_rate_limit(limit_or_interval=float(sleepytime))
 
     # Add headphones credentials
-    if headphones.MIRROR == "headphones":
+    if headphones.CONFIG.MIRROR == "headphones":
         if not mbuser and mbpass:
             logger.warn("No username or password set for VIP server")
         else:
@@ -73,6 +77,7 @@ def startmb():
 
     return True
 
+
 def findArtist(name, limit=1):
 
     with mb_lock:
@@ -123,6 +128,7 @@ def findArtist(name, limit=1):
                         })
         return artistlist
 
+
 def findRelease(name, limit=1, artist=None):
 
     with mb_lock:
@@ -201,6 +207,7 @@ def findRelease(name, limit=1, artist=None):
                         })
         return releaselist
 
+
 def getArtist(artistid, extrasonly=False):
 
     with mb_lock:
@@ -213,13 +220,13 @@ def getArtist(artistid, extrasonly=False):
             artist = musicbrainzngs.get_artist_by_id(artistid)['artist']
             newRgs = None
             artist['release-group-list'] = []
-            while newRgs == None or len(newRgs) >= limit:
+            while newRgs is None or len(newRgs) >= limit:
                 newRgs = musicbrainzngs.browse_release_groups(artistid, release_type="album", offset=len(artist['release-group-list']), limit=limit)['release-group-list']
                 artist['release-group-list'] += newRgs
         except musicbrainzngs.WebServiceError as e:
             logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
             time.sleep(5)
-        except Exception,e:
+        except Exception as e:
             pass
 
         if not artist:
@@ -247,7 +254,6 @@ def getArtist(artistid, extrasonly=False):
         #    if 'end' in artist['life-span']:
         #        artist_dict['artist_enddate'] = unicode(artist['life-span']['end'])
 
-
         releasegroups = []
 
         if not extrasonly:
@@ -278,7 +284,7 @@ def getArtist(artistid, extrasonly=False):
                 extras = map(int, db_artist['Extras'].split(','))
             else:
                 extras = []
-            extras_list = ["single", "ep", "compilation", "soundtrack", "live", "remix", "spokenword", "audiobook", "other", "dj-mix", "mixtape/street", "broadcast", "interview", "demo"]
+            extras_list = headphones.POSSIBLE_EXTRAS
 
             includes = []
 
@@ -295,7 +301,7 @@ def getArtist(artistid, extrasonly=False):
                 try:
                     limit = 200
                     newRgs = None
-                    while newRgs == None or len(newRgs) >= limit:
+                    while newRgs is None or len(newRgs) >= limit:
                         newRgs = musicbrainzngs.browse_release_groups(artistid, release_type=include, offset=len(mb_extras_list), limit=limit)['release-group-list']
                         mb_extras_list += newRgs
                 except musicbrainzngs.WebServiceError as e:
@@ -321,14 +327,13 @@ def getArtist(artistid, extrasonly=False):
 
         return artist_dict
 
+
 def getReleaseGroup(rgid):
     """
     Returns a list of releases in a release group
     """
     with mb_lock:
 
-        releaselist = []
-
         releaseGroup = None
 
         try:
@@ -342,6 +347,7 @@ def getReleaseGroup(rgid):
         else:
             return releaseGroup['release-list']
 
+
 def getRelease(releaseid, include_artist_info=True):
     """
     Deep release search to get track info
@@ -377,7 +383,6 @@ def getRelease(releaseid, include_artist_info=True):
         except:
             release['country'] = u'Unknown'
 
-
         if include_artist_info:
 
             if 'release-group' in results:
@@ -404,6 +409,7 @@ def getRelease(releaseid, include_artist_info=True):
 
         return release
 
+
 def get_new_releases(rgid, includeExtras=False, forcefull=False):
 
     myDB = db.DBConnection()
@@ -411,7 +417,7 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
     try:
         limit = 100
         newResults = None
-        while newResults == None or len(newResults) >= limit:
+        while newResults is None or len(newResults) >= limit:
             newResults = musicbrainzngs.browse_releases(release_group=rgid, includes=['artist-credits', 'labels', 'recordings', 'release-groups', 'media'], limit=limit, offset=len(results))
             if 'release-list' not in newResults:
                 break #may want to raise an exception here instead ?
@@ -457,8 +463,6 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
 
         release = {}
         rel_id_check = releasedata['id']
-        artistid = unicode(releasedata['artist-credit'][0]['artist']['id'])
-
         album_checker = myDB.action('SELECT * from allalbums WHERE ReleaseID=?', [rel_id_check]).fetchone()
         if not album_checker or forcefull:
             #DELETE all references to this release since we're updating it anyway.
@@ -486,7 +490,6 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
                 logger.warn('Release ' + releasedata['id'] + ' has no Artists associated.')
                 return False
 
-
             release['ReleaseCountry'] = unicode(releasedata['country']) if 'country' in releasedata else u'Unknown'
             #assuming that the list will contain media and that the format will be consistent
             try:
@@ -558,8 +561,6 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
 
                 myDB.upsert("alltracks", newValueDict, controlValueDict)
             num_new_releases = num_new_releases + 1
-            #print releasedata['title']
-            #print num_new_releases
             if album_checker:
                 logger.info('[%s] Existing release %s (%s) updated' % (release['ArtistName'], release['AlbumTitle'], rel_id_check))
             else:
@@ -572,6 +573,7 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
 
     return num_new_releases
 
+
 def getTracksFromRelease(release):
     totalTracks = 1
     tracks = []
@@ -592,6 +594,8 @@ def getTracksFromRelease(release):
     return tracks
 
 # Used when there is a disambiguation
+
+
 def findArtistbyAlbum(name):
 
     myDB = db.DBConnection()
@@ -615,7 +619,6 @@ def findArtistbyAlbum(name):
         logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
         time.sleep(5)
 
-
     if not results:
         return False
 
@@ -633,10 +636,9 @@ def findArtistbyAlbum(name):
         #artist_dict['url'] = u'http://musicbrainz.org/artist/' + newArtist['id']
         #artist_dict['score'] = int(releaseGroup['ext:score'])
 
-
-
     return artist_dict
 
+
 def findAlbumID(artist=None, album=None):
 
     results = None

headphones/music_encoder.py

@@ -24,18 +24,15 @@ from headphones import logger
 from beets.mediafile import MediaFile
 
 # xld
-if headphones.ENCODER == 'xld':
 import getXldProfile
-    XLD = True
+
-else:
-    XLD = False
 
 def encode(albumPath):
+    use_xld = headphones.CONFIG.ENCODER == 'xld'
 
     # Return if xld details not found
-    if XLD:
+    if use_xld:
-        global xldProfile
+        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
-        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(headphones.XLDPROFILE)
         if not xldFormat:
             logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded', xldProfile)
             return None
@@ -60,16 +57,16 @@ def encode(albumPath):
     for r, d, f in os.walk(albumPath):
         for music in f:
             if any(music.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
-                if not XLD:
+                if not use_xld:
-                    encoderFormat = headphones.ENCODEROUTPUTFORMAT.encode(headphones.SYS_ENCODING)
+                    encoderFormat = headphones.CONFIG.ENCODEROUTPUTFORMAT.encode(headphones.SYS_ENCODING)
                 else:
                     xldMusicFile = os.path.join(r, music)
                     xldInfoMusic = MediaFile(xldMusicFile)
                     encoderFormat = xldFormat
 
-                if (headphones.ENCODERLOSSLESS):
+                if (headphones.CONFIG.ENCODERLOSSLESS):
                     ext = os.path.normpath(os.path.splitext(music)[1].lstrip(".")).lower()
-                    if not XLD and ext == 'flac' or XLD and (ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
+                    if not use_xld and ext == 'flac' or use_xld and (ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
                         musicFiles.append(os.path.join(r, music))
                         musicTemp = os.path.normpath(os.path.splitext(music)[0] + '.' + encoderFormat)
                         musicTempFiles.append(os.path.join(tempDirEncode, musicTemp))
@@ -80,23 +77,23 @@ def encode(albumPath):
                     musicTemp = os.path.normpath(os.path.splitext(music)[0] + '.' + encoderFormat)
                     musicTempFiles.append(os.path.join(tempDirEncode, musicTemp))
 
-    if headphones.ENCODER_PATH:
+    if headphones.CONFIG.ENCODER_PATH:
-        encoder = headphones.ENCODER_PATH.encode(headphones.SYS_ENCODING)
+        encoder = headphones.CONFIG.ENCODER_PATH.encode(headphones.SYS_ENCODING)
     else:
-        if XLD:
+        if use_xld:
             encoder = os.path.join('/Applications', 'xld')
-        elif headphones.ENCODER =='lame':
+        elif headphones.CONFIG.ENCODER == 'lame':
             if headphones.SYS_PLATFORM == "win32":
                 ## NEED THE DEFAULT LAME INSTALL ON WIN!
                 encoder = "C:/Program Files/lame/lame.exe"
             else:
                 encoder = "lame"
-        elif headphones.ENCODER =='ffmpeg':
+        elif headphones.CONFIG.ENCODER == 'ffmpeg':
             if headphones.SYS_PLATFORM == "win32":
                 encoder = "C:/Program Files/ffmpeg/bin/ffmpeg.exe"
             else:
                 encoder = "ffmpeg"
-        elif headphones.ENCODER == 'libav':
+        elif headphones.CONFIG.ENCODER == 'libav':
             if headphones.SYS_PLATFORM == "win32":
                 encoder = "C:/Program Files/libav/bin/avconv.exe"
             else:
@@ -110,28 +107,28 @@ def encode(albumPath):
         infoMusic = MediaFile(music)
         encode = False
 
-        if XLD:
+        if use_xld:
             if xldBitrate and (infoMusic.bitrate / 1000 <= xldBitrate):
                 logger.info('%s has bitrate <= %skb, will not be re-encoded', music.decode(headphones.SYS_ENCODING, 'replace'), xldBitrate)
             else:
                 encode = True
-        elif headphones.ENCODER == 'lame':
+        elif headphones.CONFIG.ENCODER == 'lame':
             if not any(music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + x) for x in ["mp3", "wav"]):
                 logger.warn('Lame cannot encode %s format for %s, use ffmpeg', os.path.splitext(music)[1], music)
             else:
-                if (music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (int(infoMusic.bitrate / 1000) <= headphones.BITRATE)):
+                if (music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE)):
-                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.BITRATE)
+                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
                 else:
                     encode = True
         else:
-            if headphones.ENCODEROUTPUTFORMAT=='ogg':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                 if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.ogg'):
                     logger.warn('Cannot re-encode .ogg %s', music.decode(headphones.SYS_ENCODING, 'replace'))
                 else:
                     encode = True
-            elif (headphones.ENCODEROUTPUTFORMAT=='mp3' or headphones.ENCODEROUTPUTFORMAT=='m4a'):
+            elif (headphones.CONFIG.ENCODEROUTPUTFORMAT == 'mp3' or headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a'):
-                if (music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.'+headphones.ENCODEROUTPUTFORMAT) and (int(infoMusic.bitrate / 1000 ) <= headphones.BITRATE)):
+                if (music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + headphones.CONFIG.ENCODEROUTPUTFORMAT) and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE)):
-                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.BITRATE)
+                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
                 else:
                     encode = True
         # encode
@@ -149,11 +146,11 @@ def encode(albumPath):
         processes = 1
 
         # Use multicore if enabled
-        if headphones.ENCODER_MULTICORE:
+        if headphones.CONFIG.ENCODER_MULTICORE:
-            if headphones.ENCODER_MULTICORE_COUNT == 0:
+            if headphones.CONFIG.ENCODER_MULTICORE_COUNT == 0:
                 processes = multiprocessing.cpu_count()
             else:
-                processes = headphones.ENCODER_MULTICORE_COUNT
+                processes = headphones.CONFIG.ENCODER_MULTICORE_COUNT
 
             logger.debug("Multi-core encoding enabled, spawning %d processes",
                 processes)
@@ -194,14 +191,14 @@ def encode(albumPath):
         for dest in musicTempFiles:
             if os.path.exists(dest):
                 source = musicFiles[i]
-                if headphones.DELETE_LOSSLESS_FILES:
+                if headphones.CONFIG.DELETE_LOSSLESS_FILES:
                     os.remove(source)
                 check_dest = os.path.join(albumPath, os.path.split(dest)[1])
                 if os.path.exists(check_dest):
                     os.remove(check_dest)
                 try:
                     shutil.move(dest, albumPath)
-                except Exception, e:
+                except Exception as e:
                     logger.error('Could not move %s to %s: %s', dest, albumPath, e)
                     encoder_failed = True
                     break
@@ -212,7 +209,7 @@ def encode(albumPath):
 
     # Return with error if any encoding errors
     if encoder_failed:
-        logger.error("One or more files failed to encode. Ensure you have the latest version of %s installed.", headphones.ENCODER)
+        logger.error("One or more files failed to encode. Ensure you have the latest version of %s installed.", headphones.CONFIG.ENCODER)
         return None
 
     time.sleep(1)
@@ -226,6 +223,7 @@ def encode(albumPath):
 
     return musicFinalFiles
 
+
 def command_map(args):
     """
     Wrapper for the '[multiprocessing.]map()' method, to unpack the arguments
@@ -239,21 +237,27 @@ def command_map(args):
     # Start encoding
     try:
         return command(*args)
-    except Exception as e:
+    except Exception:
         logger.exception("Encoder raised an exception.")
         return False
 
+
 def command(encoder, musicSource, musicDest, albumPath):
     """
     Encode a given music file with a certain encoder. Returns True on success,
     or False otherwise.
     """
+    use_xld = headphones.CONFIG.ENCODER == 'xld'
 
     startMusicTime = time.time()
     cmd = []
 
-    # XLD
+    # Return if xld details not found
-    if XLD:
+    if use_xld:
+        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
+        if not xldFormat:
+            logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded', xldProfile)
+            return None
         xldDestDir = os.path.split(musicDest)[0]
         cmd = [encoder]
         cmd.extend([musicSource])
@@ -263,17 +267,17 @@ def command(encoder, musicSource, musicDest, albumPath):
         cmd.extend([xldDestDir])
 
     # Lame
-    elif headphones.ENCODER == 'lame':
+    elif headphones.CONFIG.ENCODER == 'lame':
         cmd = [encoder]
         opts = []
-        if not headphones.ADVANCEDENCODER:
+        if not headphones.CONFIG.ADVANCEDENCODER:
             opts.extend(['-h'])
-            if headphones.ENCODERVBRCBR=='cbr':
+            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['--resample', str(headphones.SAMPLINGFREQUENCY), '-b', str(headphones.BITRATE)])
+                opts.extend(['--resample', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-b', str(headphones.CONFIG.BITRATE)])
-            elif headphones.ENCODERVBRCBR=='vbr':
+            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
-                opts.extend(['-v', str(headphones.ENCODERQUALITY)])
+                opts.extend(['-v', str(headphones.CONFIG.ENCODERQUALITY)])
         else:
-            advanced = (headphones.ADVANCEDENCODER.split())
+            advanced = (headphones.CONFIG.ADVANCEDENCODER.split())
             for tok in advanced:
                 opts.extend([tok.encode(headphones.SYS_ENCODING)])
         opts.extend([musicSource])
@@ -281,42 +285,42 @@ def command(encoder, musicSource, musicDest, albumPath):
         cmd.extend(opts)
 
     # FFmpeg
-    elif headphones.ENCODER == 'ffmpeg':
+    elif headphones.CONFIG.ENCODER == 'ffmpeg':
         cmd = [encoder, '-i', musicSource]
         opts = []
-        if not headphones.ADVANCEDENCODER:
+        if not headphones.CONFIG.ADVANCEDENCODER:
-            if headphones.ENCODEROUTPUTFORMAT=='ogg':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                 opts.extend(['-acodec', 'libvorbis'])
-            if headphones.ENCODEROUTPUTFORMAT=='m4a':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
                 opts.extend(['-strict', 'experimental'])
-            if headphones.ENCODERVBRCBR=='cbr':
+            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['-ar', str(headphones.SAMPLINGFREQUENCY), '-ab', str(headphones.BITRATE) + 'k'])
+                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab', str(headphones.CONFIG.BITRATE) + 'k'])
-            elif headphones.ENCODERVBRCBR=='vbr':
+            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
-                opts.extend(['-aq', str(headphones.ENCODERQUALITY)])
+                opts.extend(['-aq', str(headphones.CONFIG.ENCODERQUALITY)])
             opts.extend(['-y', '-ac', '2', '-vn'])
         else:
-            advanced = (headphones.ADVANCEDENCODER.split())
+            advanced = (headphones.CONFIG.ADVANCEDENCODER.split())
             for tok in advanced:
                 opts.extend([tok.encode(headphones.SYS_ENCODING)])
         opts.extend([musicDest])
         cmd.extend(opts)
 
     # Libav
-    elif headphones.ENCODER == "libav":
+    elif headphones.CONFIG.ENCODER == "libav":
         cmd = [encoder, '-i', musicSource]
         opts = []
-        if not headphones.ADVANCEDENCODER:
+        if not headphones.CONFIG.ADVANCEDENCODER:
-            if headphones.ENCODEROUTPUTFORMAT=='ogg':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                 opts.extend(['-acodec', 'libvorbis'])
-            if headphones.ENCODEROUTPUTFORMAT=='m4a':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
                 opts.extend(['-strict', 'experimental'])
-            if headphones.ENCODERVBRCBR=='cbr':
+            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['-ar', str(headphones.SAMPLINGFREQUENCY), '-ab', str(headphones.BITRATE) + 'k'])
+                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab', str(headphones.CONFIG.BITRATE) + 'k'])
-            elif headphones.ENCODERVBRCBR=='vbr':
+            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
-                opts.extend(['-aq', str(headphones.ENCODERQUALITY)])
+                opts.extend(['-aq', str(headphones.CONFIG.ENCODERQUALITY)])
             opts.extend(['-y', '-ac', '2', '-vn'])
         else:
-            advanced = (headphones.ADVANCEDENCODER.split())
+            advanced = (headphones.CONFIG.ADVANCEDENCODER.split())
             for tok in advanced:
                 opts.extend([tok.encode(headphones.SYS_ENCODING)])
         opts.extend([musicDest])
@@ -339,7 +343,7 @@ def command(encoder, musicSource, musicDest, albumPath):
     process = subprocess.Popen(cmd, startupinfo=startupinfo,
         stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
         stderr=subprocess.PIPE)
-    stdout, stderr = process.communicate(headphones.ENCODER)
+    stdout, stderr = process.communicate(headphones.CONFIG.ENCODER)
 
     # Error if return code not zero
     if process.returncode:
@@ -347,7 +351,7 @@ def command(encoder, musicSource, musicDest, albumPath):
         out = stdout if stdout else stderr
         out = out.decode(headphones.SYS_ENCODING, 'replace')
         outlast2lines = '\n'.join(out.splitlines()[-2:])
-        logger.error('%s error details: %s' % (headphones.ENCODER, outlast2lines))
+        logger.error('%s error details: %s' % (headphones.CONFIG.ENCODER, outlast2lines))
         out = out.rstrip("\n")
         logger.debug(out)
         encoded = False
@@ -357,6 +361,7 @@ def command(encoder, musicSource, musicDest, albumPath):
 
     return encoded
 
+
 def getTimeEncode(start):
     seconds = int(time.time() - start)
     hours = seconds / 3600

headphones/notifiers.py

@@ -28,7 +28,6 @@ import headphones
 import os.path
 import subprocess
 import gntp.notifier
-import time
 import json
 
 import oauth2 as oauth
@@ -39,15 +38,16 @@ try:
 except ImportError:
     from cgi import parse_qsl
 
+
 class GROWL(object):
     """
-    Growl notifications, for OS X
+    Growl notifications, for OS X.
     """
 
     def __init__(self):
-        self.enabled = headphones.GROWL_ENABLED
+        self.enabled = headphones.CONFIG.GROWL_ENABLED
-        self.host = headphones.GROWL_HOST
+        self.host = headphones.CONFIG.GROWL_HOST
-        self.password = headphones.GROWL_PASSWORD
+        self.password = headphones.CONFIG.GROWL_PASSWORD
 
     def conf(self, options):
         return cherrypy.config['config'].get('Growl', options)
@@ -84,14 +84,18 @@ class GROWL(object):
         try:
             growl.register()
         except gntp.notifier.errors.NetworkError:
-            logger.info(u'Growl notification failed: network error')
+            logger.warning(u'Growl notification failed: network error')
             return
         except gntp.notifier.errors.AuthError:
-            logger.info(u'Growl notification failed: authentication error')
+            logger.warning(u'Growl notification failed: authentication error')
             return
 
+        # Fix message
+        message = message.encode(headphones.SYS_ENCODING, "replace")
+
         # Send it, including an image
-        image_file = os.path.join(str(headphones.PROG_DIR), 'data/images/headphoneslogo.png')
+        image_file = os.path.join(str(headphones.PROG_DIR),
+            "data/images/headphoneslogo.png")
 
         with open(image_file, 'rb') as f:
             image = f.read()
@@ -104,7 +108,7 @@ class GROWL(object):
                 icon=image
             )
         except gntp.notifier.errors.NetworkError:
-            logger.info(u'Growl notification failed: network error')
+            logger.warning(u'Growl notification failed: network error')
             return
 
         logger.info(u"Growl notifications sent.")
@@ -120,30 +124,31 @@ class GROWL(object):
 
         self.notify('ZOMG Lazors Pewpewpew!', 'Test Message')
 
+
 class PROWL(object):
     """
     Prowl notifications.
     """
 
     def __init__(self):
-        self.enabled = headphones.PROWL_ENABLED
+        self.enabled = headphones.CONFIG.PROWL_ENABLED
-        self.keys = headphones.PROWL_KEYS
+        self.keys = headphones.CONFIG.PROWL_KEYS
-        self.priority = headphones.PROWL_PRIORITY
+        self.priority = headphones.CONFIG.PROWL_PRIORITY
 
     def conf(self, options):
         return cherrypy.config['config'].get('Prowl', options)
 
     def notify(self, message, event):
-        if not headphones.PROWL_ENABLED:
+        if not headphones.CONFIG.PROWL_ENABLED:
             return
 
         http_handler = HTTPSConnection("api.prowlapp.com")
 
-        data = {'apikey': headphones.PROWL_KEYS,
+        data = {'apikey': headphones.CONFIG.PROWL_KEYS,
                 'application': 'Headphones',
                 'event': event,
                 'description': message.encode("utf-8"),
-                'priority': headphones.PROWL_PRIORITY }
+                'priority': headphones.CONFIG.PROWL_PRIORITY}
 
         http_handler.request("POST",
                                 "/publicapi/add",
@@ -173,6 +178,7 @@ class PROWL(object):
 
         self.notify('ZOMG Lazors Pewpewpew!', 'Test Message')
 
+
 class MPC(object):
     """
     MPC library update
@@ -193,9 +199,9 @@ class XBMC(object):
 
     def __init__(self):
 
-        self.hosts = headphones.XBMC_HOST
+        self.hosts = headphones.CONFIG.XBMC_HOST
-        self.username = headphones.XBMC_USERNAME
+        self.username = headphones.CONFIG.XBMC_USERNAME
-        self.password = headphones.XBMC_PASSWORD
+        self.password = headphones.CONFIG.XBMC_PASSWORD
 
     def _sendhttp(self, host, command):
         url_command = urllib.urlencode(command)
@@ -260,13 +266,14 @@ class XBMC(object):
             except Exception:
                 logger.error('Error sending notification request to XBMC')
 
+
 class LMS(object):
     """
     Class for updating a Logitech Media Server
     """
 
     def __init__(self):
-        self.hosts = headphones.LMS_HOST
+        self.hosts = headphones.CONFIG.LMS_HOST
 
     def _sendjson(self, host):
         data = {'id': 1, 'method': 'slim.request', 'params': ["", ["rescan"]]}
@@ -278,7 +285,7 @@ class LMS(object):
 
         try:
             handle = urllib2.urlopen(req)
-        except Exception, e:
+        except Exception as e:
             logger.warn('Error opening LMS url: %s' % e)
             return
 
@@ -301,13 +308,14 @@ class LMS(object):
             if not request:
                 logger.warn('Error sending rescan request to LMS')
 
+
 class Plex(object):
     def __init__(self):
 
-        self.server_hosts = headphones.PLEX_SERVER_HOST
+        self.server_hosts = headphones.CONFIG.PLEX_SERVER_HOST
-        self.client_hosts = headphones.PLEX_CLIENT_HOST
+        self.client_hosts = headphones.CONFIG.PLEX_CLIENT_HOST
-        self.username = headphones.PLEX_USERNAME
+        self.username = headphones.CONFIG.PLEX_USERNAME
-        self.password = headphones.PLEX_PASSWORD
+        self.password = headphones.CONFIG.PLEX_PASSWORD
 
     def _sendhttp(self, host, command):
 
@@ -328,7 +336,7 @@ class Plex(object):
 
         try:
             handle = urllib2.urlopen(req)
-        except Exception, e:
+        except Exception as e:
             logger.warn('Error opening Plex url: %s' % e)
             return
 
@@ -362,7 +370,7 @@ class Plex(object):
                     url = "%s/library/sections/%s/refresh" % (host, s.getAttribute('key'))
                     try:
                         urllib.urlopen(url)
-                    except Exception, e:
+                    except Exception as e:
                         logger.warn("Error updating library section for Plex Media Server: %s" % e)
                         return False
 
@@ -387,11 +395,12 @@ class Plex(object):
             except:
                 logger.warn('Error sending notification request to Plex Media Server')
 
+
 class NMA(object):
     def notify(self, artist=None, album=None, snatched=None):
         title = 'Headphones'
-        api = headphones.NMA_APIKEY
+        api = headphones.CONFIG.NMA_APIKEY
-        nma_priority = headphones.NMA_PRIORITY
+        nma_priority = headphones.CONFIG.NMA_PRIORITY
 
         logger.debug(u"NMA title: " + title)
         logger.debug(u"NMA API: " + api)
@@ -413,7 +422,8 @@ class NMA(object):
         keys = api.split(',')
         p.addkey(keys)
 
-        if len(keys) > 1: batch = True
+        if len(keys) > 1:
+            batch = True
 
         response = p.push(title, event, message, priority=nma_priority, batch_mode=batch)
 
@@ -423,22 +433,23 @@ class NMA(object):
         else:
             return True
 
+
 class PUSHBULLET(object):
 
     def __init__(self):
-        self.apikey = headphones.PUSHBULLET_APIKEY
+        self.apikey = headphones.CONFIG.PUSHBULLET_APIKEY
-        self.deviceid = headphones.PUSHBULLET_DEVICEID
+        self.deviceid = headphones.CONFIG.PUSHBULLET_DEVICEID
 
     def conf(self, options):
         return cherrypy.config['config'].get('PUSHBULLET', options)
 
     def notify(self, message, event):
-        if not headphones.PUSHBULLET_ENABLED:
+        if not headphones.CONFIG.PUSHBULLET_ENABLED:
             return
 
         http_handler = HTTPSConnection("api.pushbullet.com")
 
-        data = {'device_iden': headphones.PUSHBULLET_DEVICEID,
+        data = {'device_iden': headphones.CONFIG.PUSHBULLET_DEVICEID,
                 'type': "note",
                 'title': "Headphones",
                 'body': message.encode("utf-8")}
@@ -446,7 +457,7 @@ class PUSHBULLET(object):
         http_handler.request("POST",
                                 "/api/pushes",
                                 headers={'Content-type': "application/x-www-form-urlencoded",
-                                            'Authorization' : 'Basic %s' % base64.b64encode(headphones.PUSHBULLET_APIKEY + ":") },
+                                            'Authorization': 'Basic %s' % base64.b64encode(headphones.CONFIG.PUSHBULLET_APIKEY + ":")},
                                 body=urlencode(data))
         response = http_handler.getresponse()
         request_status = response.status
@@ -476,13 +487,14 @@ class PUSHBULLET(object):
 
         self.notify('Main Screen Activate', 'Test Message')
 
+
 class PUSHALOT(object):
 
     def notify(self, message, event):
-        if not headphones.PUSHALOT_ENABLED:
+        if not headphones.CONFIG.PUSHALOT_ENABLED:
             return
 
-        pushalot_authorizationtoken = headphones.PUSHALOT_APIKEY
+        pushalot_authorizationtoken = headphones.CONFIG.PUSHALOT_APIKEY
 
         logger.debug(u"Pushalot event: " + event)
         logger.debug(u"Pushalot message: " + message)
@@ -515,6 +527,7 @@ class PUSHALOT(object):
                 logger.info(u"Pushalot notification failed.")
                 return False
 
+
 class Synoindex(object):
     def __init__(self, util_loc='/usr/syno/bin/synoindex'):
         self.util_loc = util_loc
@@ -551,15 +564,16 @@ class Synoindex(object):
             for path in path_list:
                 self.notify(path)
 
+
 class PUSHOVER(object):
 
     def __init__(self):
-        self.enabled = headphones.PUSHOVER_ENABLED
+        self.enabled = headphones.CONFIG.PUSHOVER_ENABLED
-        self.keys = headphones.PUSHOVER_KEYS
+        self.keys = headphones.CONFIG.PUSHOVER_KEYS
-        self.priority = headphones.PUSHOVER_PRIORITY
+        self.priority = headphones.CONFIG.PUSHOVER_PRIORITY
 
-        if headphones.PUSHOVER_APITOKEN:
+        if headphones.CONFIG.PUSHOVER_APITOKEN:
-            self.application_token = headphones.PUSHOVER_APITOKEN
+            self.application_token = headphones.CONFIG.PUSHOVER_APITOKEN
         else:
             self.application_token = "LdPCoy0dqC21ktsbEyAVCcwvQiVlsz"
 
@@ -567,16 +581,16 @@ class PUSHOVER(object):
         return cherrypy.config['config'].get('Pushover', options)
 
     def notify(self, message, event):
-        if not headphones.PUSHOVER_ENABLED:
+        if not headphones.CONFIG.PUSHOVER_ENABLED:
             return
 
         http_handler = HTTPSConnection("api.pushover.net")
 
         data = {'token': self.application_token,
-                'user': headphones.PUSHOVER_KEYS,
+                'user': headphones.CONFIG.PUSHOVER_KEYS,
                 'title': event,
                 'message': message.encode("utf-8"),
-                'priority': headphones.PUSHOVER_PRIORITY }
+                'priority': headphones.CONFIG.PUSHOVER_PRIORITY}
 
         http_handler.request("POST",
                                 "/1/messages.json",
@@ -609,6 +623,7 @@ class PUSHOVER(object):
 
         self.notify('Main Screen Activate', 'Test Message')
 
+
 class TwitterNotifier(object):
 
     REQUEST_TOKEN_URL = 'https://api.twitter.com/oauth/request_token'
@@ -621,11 +636,11 @@ class TwitterNotifier(object):
         self.consumer_secret = "A4Xkw9i5SjHbTk7XT8zzOPqivhj9MmRDR9Qn95YA9sk"
 
     def notify_snatch(self, title):
-        if headphones.TWITTER_ONSNATCH:
+        if headphones.CONFIG.TWITTER_ONSNATCH:
             self._notifyTwitter(common.notifyStrings[common.NOTIFY_SNATCH] + ': ' + title + ' at ' + helpers.now())
 
     def notify_download(self, title):
-        if headphones.TWITTER_ENABLED:
+        if headphones.CONFIG.TWITTER_ENABLED:
             self._notifyTwitter(common.notifyStrings[common.NOTIFY_DOWNLOAD] + ': ' + title + ' at ' + helpers.now())
 
     def test_notify(self):
@@ -633,7 +648,6 @@ class TwitterNotifier(object):
 
     def _get_authorization(self):
 
-        signature_method_hmac_sha1 = oauth.SignatureMethod_HMAC_SHA1() #@UnusedVariable
         oauth_consumer = oauth.Consumer(key=self.consumer_key, secret=self.consumer_secret)
         oauth_client = oauth.Client(oauth_consumer)
 
@@ -646,16 +660,16 @@ class TwitterNotifier(object):
         else:
             request_token = dict(parse_qsl(content))
 
-            headphones.TWITTER_USERNAME = request_token['oauth_token']
+            headphones.CONFIG.TWITTER_USERNAME = request_token['oauth_token']
-            headphones.TWITTER_PASSWORD = request_token['oauth_token_secret']
+            headphones.CONFIG.TWITTER_PASSWORD = request_token['oauth_token_secret']
 
             return self.AUTHORIZATION_URL + "?oauth_token=" + request_token['oauth_token']
 
     def _get_credentials(self, key):
         request_token = {}
 
-        request_token['oauth_token'] = headphones.TWITTER_USERNAME
+        request_token['oauth_token'] = headphones.CONFIG.TWITTER_USERNAME
-        request_token['oauth_token_secret'] = headphones.TWITTER_PASSWORD
+        request_token['oauth_token_secret'] = headphones.CONFIG.TWITTER_PASSWORD
         request_token['oauth_callback_confirmed'] = 'true'
 
         token = oauth.Token(request_token['oauth_token'], request_token['oauth_token_secret'])
@@ -663,7 +677,6 @@ class TwitterNotifier(object):
 
         logger.info('Generating and signing request for an access token using key ' + key)
 
-        signature_method_hmac_sha1 = oauth.SignatureMethod_HMAC_SHA1() #@UnusedVariable
         oauth_consumer = oauth.Consumer(key=self.consumer_key, secret=self.consumer_secret)
         logger.info('oauth_consumer: ' + str(oauth_consumer))
         oauth_client = oauth.Client(oauth_consumer, token)
@@ -681,17 +694,16 @@ class TwitterNotifier(object):
         else:
             logger.info('Your Twitter Access Token key: %s' % access_token['oauth_token'])
             logger.info('Access Token secret: %s' % access_token['oauth_token_secret'])
-            headphones.TWITTER_USERNAME = access_token['oauth_token']
+            headphones.CONFIG.TWITTER_USERNAME = access_token['oauth_token']
-            headphones.TWITTER_PASSWORD = access_token['oauth_token_secret']
+            headphones.CONFIG.TWITTER_PASSWORD = access_token['oauth_token_secret']
             return True
 
-
     def _send_tweet(self, message=None):
 
         username = self.consumer_key
         password = self.consumer_secret
-        access_token_key=headphones.TWITTER_USERNAME
+        access_token_key = headphones.CONFIG.TWITTER_USERNAME
-        access_token_secret=headphones.TWITTER_PASSWORD
+        access_token_secret = headphones.CONFIG.TWITTER_PASSWORD
 
         logger.info(u"Sending tweet: " + message)
 
@@ -699,20 +711,21 @@ class TwitterNotifier(object):
 
         try:
             api.PostUpdate(message)
-        except Exception, e:
+        except Exception as e:
             logger.info(u"Error Sending Tweet: %s" % e)
             return False
 
         return True
 
     def _notifyTwitter(self, message='', force=False):
-        prefix = headphones.TWITTER_PREFIX
+        prefix = headphones.CONFIG.TWITTER_PREFIX
 
-        if not headphones.TWITTER_ENABLED and not force:
+        if not headphones.CONFIG.TWITTER_ENABLED and not force:
             return False
 
         return self._send_tweet(prefix + ": " + message)
 
+
 class OSX_NOTIFY(object):
 
     def __init__(self):
@@ -723,6 +736,7 @@ class OSX_NOTIFY(object):
 
     def swizzle(self, cls, SEL, func):
         old_IMP = cls.instanceMethodForSelector_(SEL)
+
         def wrapper(self, *args, **kwargs):
             return func(self, old_IMP, *args, **kwargs)
         new_IMP = self.objc.selector(wrapper, selector=old_IMP.selector,
@@ -761,13 +775,14 @@ class OSX_NOTIFY(object):
             del pool
             return True
 
-        except Exception, e:
+        except Exception as e:
             logger.warn('Error sending OS X Notification: %s' % e)
             return False
 
     def swizzled_bundleIdentifier(self, original, swizzled):
         return 'ade.headphones.osxnotify'
 
+
 class BOXCAR(object):
 
     def __init__(self):
@@ -779,7 +794,7 @@ class BOXCAR(object):
                 message += '<br></br><a href="http://musicbrainz.org/release-group/%s">MusicBrainz</a>' % rgid
 
             data = urllib.urlencode({
-                'user_credentials': headphones.BOXCAR_TOKEN,
+                'user_credentials': headphones.CONFIG.BOXCAR_TOKEN,
                 'notification[title]': title.encode('utf-8'),
                 'notification[long_message]': message.encode('utf-8'),
                 'notification[sound]': "done"
@@ -794,12 +809,13 @@ class BOXCAR(object):
             logger.warn('Error sending Boxcar2 Notification: %s' % e)
             return False
 
+
 class SubSonicNotifier(object):
 
     def __init__(self):
-        self.host = headphones.SUBSONIC_HOST
+        self.host = headphones.CONFIG.SUBSONIC_HOST
-        self.username = headphones.SUBSONIC_USERNAME
+        self.username = headphones.CONFIG.SUBSONIC_USERNAME
-        self.password = headphones.SUBSONIC_PASSWORD
+        self.password = headphones.CONFIG.SUBSONIC_PASSWORD
 
     def notify(self, albumpaths):
         # Correct URL

headphones/nzbget.py

@@ -19,37 +19,33 @@
 # along with Sick Beard.  If not, see <http://www.gnu.org/licenses/>.
 
 
-
 import httplib
-import datetime
 
 import headphones
 
 from base64 import standard_b64encode
 import xmlrpclib
 
-#from headphones.providers.generic import GenericProvider
-
 from headphones import logger
 
+
 def sendNZB(nzb):
 
     addToTop = False
     nzbgetXMLrpc = "%(username)s:%(password)s@%(host)s/xmlrpc"
 
-    if headphones.NZBGET_HOST == None:
+    if headphones.CONFIG.NZBGET_HOST is None:
         logger.error(u"No NZBget host found in configuration. Please configure it.")
         return False
 
-    if headphones.NZBGET_HOST.startswith('https://'):
+    if headphones.CONFIG.NZBGET_HOST.startswith('https://'):
         nzbgetXMLrpc = 'https://' + nzbgetXMLrpc
-        headphones.NZBGET_HOST.replace('https://','',1)
+        headphones.CONFIG.NZBGET_HOST.replace('https://', '', 1)
     else:
         nzbgetXMLrpc = 'http://' + nzbgetXMLrpc
-        headphones.NZBGET_HOST.replace('http://','',1)
+        headphones.CONFIG.NZBGET_HOST.replace('http://', '', 1)
 
-
+    url = nzbgetXMLrpc % {"host": headphones.CONFIG.NZBGET_HOST, "username": headphones.CONFIG.NZBGET_USERNAME, "password": headphones.CONFIG.NZBGET_PASSWORD}
-    url = nzbgetXMLrpc % {"host": headphones.NZBGET_HOST, "username": headphones.NZBGET_USERNAME, "password": headphones.NZBGET_PASSWORD}
 
     nzbGetRPC = xmlrpclib.ServerProxy(url)
     try:
@@ -86,35 +82,37 @@ def sendNZB(nzb):
         nzbget_version = int(nzbget_version_str[:nzbget_version_str.find(".")])
         if nzbget_version == 0:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, addToTop, nzbcontent64)
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, addToTop, nzbcontent64)
             else:
-                if nzb.resultType == "nzb":
+                # from headphones.common.providers.generic import GenericProvider
-                    genProvider = GenericProvider("")
+                # if nzb.resultType == "nzb":
-                    data = genProvider.getURL(nzb.url)
+                #     genProvider = GenericProvider("")
-                    if (data == None):
+                #     data = genProvider.getURL(nzb.url)
+                #     if (data is None):
+                #         return False
+                #     nzbcontent64 = standard_b64encode(data)
+                # nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, addToTop, nzbcontent64)
                 return False
-                    nzbcontent64 = standard_b64encode(data)
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, addToTop, nzbcontent64)
         elif nzbget_version == 12:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
                                                  nzbcontent64, False, dupekey, dupescore, "score")
             else:
-                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
                                                     nzb.url, False, dupekey, dupescore, "score")
         # v13+ has a new combined append method that accepts both (url and content)
         # also the return value has changed from boolean to integer
         # (Positive number representing NZBID of the queue item. 0 and negative numbers represent error codes.)
         elif nzbget_version >= 13:
             nzbget_result = True if nzbGetRPC.append(nzb.name + ".nzb", nzbcontent64 if nzbcontent64 is not None else nzb.url,
-                                                     headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False, False, dupekey, dupescore,
+                                                     headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False, False, dupekey, dupescore,
                                                      "score") > 0 else False
         else:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
                                                  nzbcontent64)
             else:
-                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
                                                     nzb.url)
 
         if nzbget_result:

headphones/postprocessor.py

@@ -23,13 +23,15 @@ import headphones
 
 from beets import autotag
 from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
+from beetsplug import lyrics as beetslyrics
 
 from headphones import notifiers, utorrent, transmission
-from headphones import db, albumart, librarysync, lyrics
+from headphones import db, albumart, librarysync
 from headphones import logger, helpers, request, mb, music_encoder
 
 postprocessor_lock = threading.Lock()
 
+
 def checkFolder():
 
     with postprocessor_lock:
@@ -42,9 +44,9 @@ def checkFolder():
             if album['FolderName']:
 
                 if album['Kind'] == 'nzb':
-                    download_dir = headphones.DOWNLOAD_DIR
+                    download_dir = headphones.CONFIG.DOWNLOAD_DIR
                 else:
-                    download_dir = headphones.DOWNLOAD_TORRENT_DIR
+                    download_dir = headphones.CONFIG.DOWNLOAD_TORRENT_DIR
 
                 album_path = os.path.join(download_dir, album['FolderName']).encode(headphones.SYS_ENCODING, 'replace')
                 logger.info("Checking if %s exists" % album_path)
@@ -55,6 +57,7 @@ def checkFolder():
             else:
                 logger.info("No folder name found for " + album['Title'])
 
+
 def verify(albumid, albumpath, Kind=None, forced=False):
 
     myDB = db.DBConnection()
@@ -67,7 +70,7 @@ def verify(albumid, albumpath, Kind=None, forced=False):
         # Fetch album information from MusicBrainz
         try:
             release_list = mb.getReleaseGroup(albumid)
-        except Exception, e:
+        except Exception as e:
             logger.error('Unable to get release information for manual album with rgid: %s. Error: %s', albumid, e)
             return
 
@@ -88,7 +91,7 @@ def verify(albumid, albumpath, Kind=None, forced=False):
         # frozen during post processing, new artists will not be processed. This
         # prevents new artists from appearing suddenly. In case forced is True,
         # this check is skipped, since it is assumed the user wants this.
-        if headphones.FREEZE_DB and not forced:
+        if headphones.CONFIG.FREEZE_DB and not forced:
             artist = myDB.select("SELECT ArtistName, ArtistID FROM artists WHERE ArtistId=? OR ArtistName=?", [release_dict['artist_id'], release_dict['artist_name']])
 
             if not artist:
@@ -113,9 +116,9 @@ def verify(albumid, albumpath, Kind=None, forced=False):
 
         logger.info("ArtistID: " + release_dict['artist_id'] + " , ArtistName: " + release_dict['artist_name'])
 
-        if headphones.INCLUDE_EXTRAS:
+        if headphones.CONFIG.INCLUDE_EXTRAS:
             newValueDict['IncludeExtras'] = 1
-            newValueDict['Extras'] = headphones.EXTRAS
+            newValueDict['Extras'] = headphones.CONFIG.EXTRAS
 
         myDB.upsert("artists", newValueDict, controlValueDict)
 
@@ -176,66 +179,18 @@ def verify(albumid, albumpath, Kind=None, forced=False):
                 logger.info("Looks like " + os.path.basename(albumpath).decode(headphones.SYS_ENCODING, 'replace') + " isn't complete yet. Will try again on the next run")
                 return
 
-
+    # Split cue
-    # use xld to split cue
+    if headphones.CONFIG.CUE_SPLIT and downloaded_cuecount and downloaded_cuecount >= len(downloaded_track_list):
-
+        if headphones.CONFIG.KEEP_TORRENT_FILES and Kind == "torrent":
-    if headphones.ENCODER == 'xld' and headphones.MUSIC_ENCODER and downloaded_cuecount and downloaded_cuecount >= len(downloaded_track_list):
+            albumpath = helpers.preserve_torrent_direcory(albumpath)
-
+        if albumpath and helpers.cue_split(albumpath):
-        import getXldProfile
+            downloaded_track_list = helpers.get_downloaded_track_list(albumpath)
-
-        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(headphones.XLDPROFILE)
-        if not xldFormat:
-            logger.info(u'Details for xld profile "%s" not found, cannot split cue' % (xldProfile))
         else:
-            if headphones.ENCODERFOLDER:
+            myDB.action('UPDATE snatched SET status = "Unprocessed" WHERE status NOT LIKE "Seed%" and AlbumID=?', [albumid])
-                xldencoder = os.path.join(headphones.ENCODERFOLDER, 'xld')
+            processed = re.search(r' \(Unprocessed\)(?:\[\d+\])?', albumpath)
-            else:
+            if not processed:
-                xldencoder = os.path.join('/Applications','xld')
+                renameUnprocessedFolder(albumpath)
-
+            return
-            for r,d,f in os.walk(albumpath):
-                xldfolder = r
-                xldfile = ''
-                xldcue = ''
-                for file in f:
-                    if any(file.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS) and not xldfile:
-                        xldfile = os.path.join(r, file)
-                    elif file.lower().endswith('.cue') and not xldcue:
-                        xldcue = os.path.join(r, file)
-
-                if xldfile and xldcue and xldfolder:
-                    xldcmd = xldencoder
-                    xldcmd = xldcmd + ' "' + xldfile + '"'
-                    xldcmd = xldcmd + ' -c'
-                    xldcmd = xldcmd + ' "' + xldcue + '"'
-                    xldcmd = xldcmd + ' --profile'
-                    xldcmd = xldcmd + ' "' + xldProfile + '"'
-                    xldcmd = xldcmd + ' -o'
-                    xldcmd = xldcmd + ' "' + xldfolder + '"'
-                    logger.info(u"Cue found, splitting file " + xldfile.decode(headphones.SYS_ENCODING, 'replace'))
-                    logger.debug(xldcmd)
-                    os.system(xldcmd)
-
-            # count files, should now be more than original if xld successfully split
-
-            new_downloaded_track_list_count = 0
-            for r,d,f in os.walk(albumpath):
-                for file in f:
-                    if any(file.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
-                        new_downloaded_track_list_count += 1
-
-            if new_downloaded_track_list_count > len(downloaded_track_list):
-
-                # rename original unsplit files
-                for downloaded_track in downloaded_track_list:
-                    os.rename(downloaded_track, downloaded_track + '.original')
-
-                #reload
-
-                downloaded_track_list = []
-                for r,d,f in os.walk(albumpath):
-                    for file in f:
-                        if any(file.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
-                            downloaded_track_list.append(os.path.join(r, file))
 
     # test #1: metadata - usually works
     logger.debug('Verifying metadata...')
@@ -243,7 +198,7 @@ def verify(albumid, albumpath, Kind=None, forced=False):
     for downloaded_track in downloaded_track_list:
         try:
             f = MediaFile(downloaded_track)
-        except Exception, e:
+        except Exception as e:
             logger.info(u"Exception from MediaFile for: " + downloaded_track.decode(headphones.SYS_ENCODING, 'replace') + u" : " + unicode(e))
             continue
 
@@ -322,18 +277,19 @@ def verify(albumid, albumpath, Kind=None, forced=False):
     else:
         logger.info(u"Already marked as unprocessed: " + albumpath.decode(headphones.SYS_ENCODING, 'replace'))
 
+
 def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind=None):
 
     logger.info('Starting post-processing for: %s - %s' % (release['ArtistName'], release['AlbumTitle']))
     # Check to see if we're preserving the torrent dir
-    if headphones.KEEP_TORRENT_FILES and Kind=="torrent":
+    if headphones.CONFIG.KEEP_TORRENT_FILES and Kind == "torrent" and 'headphones-modified' not in albumpath:
         new_folder = os.path.join(albumpath, 'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
         logger.info("Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")
         try:
             shutil.copytree(albumpath, new_folder)
             # Update the album path with the new location
             albumpath = new_folder
-        except Exception, e:
+        except Exception as e:
             logger.warn("Cannot copy/move files to temp folder: " + new_folder.decode(headphones.SYS_ENCODING, 'replace') + ". Not continuing. Error: " + str(e))
             return
 
@@ -341,37 +297,48 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
         # Could probably just throw in the "headphones-modified" folder,
         # but this is good to make sure we're not counting files that may have failed to move
         downloaded_track_list = []
-        downloaded_cuecount = 0
 
         for r, d, f in os.walk(albumpath):
             for files in f:
                 if any(files.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
                     downloaded_track_list.append(os.path.join(r, files))
-                elif files.lower().endswith('.cue'):
-                    downloaded_cuecount += 1
 
     # Check if files are valid media files and are writeable, before the steps
     # below are executed. This simplifies errors and prevents unfinished steps.
     for downloaded_track in downloaded_track_list:
         try:
-            media_file = MediaFile(downloaded_track)
+            f = MediaFile(downloaded_track)
+            if f is None:
+                # this test is just to keep pyflakes from complaining about an unused variable
+                return
         except (FileTypeError, UnreadableFileError):
-            logger.error("Track file is not a valid media file: %s. Not continuing.", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
+            logger.error("Track file is not a valid media file: %s. Not " \
+                "continuing.", downloaded_track.decode(
+                    headphones.SYS_ENCODING, "replace"))
+            return
+        except IOError:
+            logger.error("Unable to find media file: %s. Not continuing.")
             return
 
-        # Not sure if line(s) below are needed, since it is possible to not
+        # If one of the options below is set, it will access/touch/modify the
-        # touch any files.
+        # files, which requires write permissions. This step just check this, so
-        if headphones.EMBED_ALBUM_ART or headphones.CLEANUP_FILES or \
+        # it will not try and fail lateron, with strange exceptions.
-           headphones.ADD_ALBUM_ART or headphones.CORRECT_METADATA or \
+        if headphones.CONFIG.EMBED_ALBUM_ART or headphones.CONFIG.CLEANUP_FILES or \
-           headphones.EMBED_LYRICS or headphones.RENAME_FILES or \
+           headphones.CONFIG.ADD_ALBUM_ART or headphones.CONFIG.CORRECT_METADATA or \
-           headphones.MOVE_FILES:
+           headphones.CONFIG.EMBED_LYRICS or headphones.CONFIG.RENAME_FILES or \
+           headphones.CONFIG.MOVE_FILES:
 
-            if not os.access(downloaded_track, os.W_OK):
+            try:
-                logger.error("Track file is not writeable, which is required for some post processing steps: %s", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
+                with open(downloaded_track, "a+b"):
+                    pass
+            except IOError:
+                logger.error("Track file is not writeable. This is required " \
+                    "for some post processing steps: %s. Not continuing.",
+                    downloaded_track.decode(headphones.SYS_ENCODING, "replace"))
                 return
 
     #start encoding
-    if headphones.MUSIC_ENCODER:
+    if headphones.CONFIG.MUSIC_ENCODER:
         downloaded_track_list = music_encoder.encode(albumpath)
 
         if not downloaded_track_list:
@@ -379,7 +346,7 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
 
     artwork = None
     album_art_path = albumart.getAlbumArt(albumid)
-    if headphones.EMBED_ALBUM_ART or headphones.ADD_ALBUM_ART:
+    if headphones.CONFIG.EMBED_ALBUM_ART or headphones.CONFIG.ADD_ALBUM_ART:
 
         if album_art_path:
             artwork = request.request_content(album_art_path)
@@ -393,31 +360,31 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
                 artwork = False
                 logger.info("No suitable album art found from Last.FM. Not adding album art")
 
-    if headphones.EMBED_ALBUM_ART and artwork:
+    if headphones.CONFIG.EMBED_ALBUM_ART and artwork:
         embedAlbumArt(artwork, downloaded_track_list)
 
-    if headphones.CLEANUP_FILES:
+    if headphones.CONFIG.CLEANUP_FILES:
         cleanupFiles(albumpath)
 
-    if headphones.KEEP_NFO:
+    if headphones.CONFIG.KEEP_NFO:
         renameNFO(albumpath)
 
-    if headphones.ADD_ALBUM_ART and artwork:
+    if headphones.CONFIG.ADD_ALBUM_ART and artwork:
         addAlbumArt(artwork, albumpath, release)
 
-    if headphones.CORRECT_METADATA:
+    if headphones.CONFIG.CORRECT_METADATA:
         correctMetadata(albumid, release, downloaded_track_list)
 
-    if headphones.EMBED_LYRICS:
+    if headphones.CONFIG.EMBED_LYRICS:
         embedLyrics(downloaded_track_list)
 
-    if headphones.RENAME_FILES:
+    if headphones.CONFIG.RENAME_FILES:
         renameFiles(albumpath, downloaded_track_list, release)
 
-    if headphones.MOVE_FILES and not headphones.DESTINATION_DIR:
+    if headphones.CONFIG.MOVE_FILES and not headphones.CONFIG.DESTINATION_DIR:
         logger.error('No DESTINATION_DIR has been set. Set "Destination Directory" to the parent directory you want to move the files to')
         albumpaths = [albumpath]
-    elif headphones.MOVE_FILES and headphones.DESTINATION_DIR:
+    elif headphones.CONFIG.MOVE_FILES and headphones.CONFIG.DESTINATION_DIR:
         albumpaths = moveFiles(albumpath, release, tracks)
     else:
         albumpaths = [albumpath]
@@ -429,13 +396,13 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
     myDB.action('UPDATE snatched SET status = "Processed" WHERE Status NOT LIKE "Seed%" and AlbumID=?', [albumid])
 
     # Check if torrent has finished seeding
-    if headphones.TORRENT_DOWNLOADER == 1 or headphones.TORRENT_DOWNLOADER == 2:
+    if headphones.CONFIG.TORRENT_DOWNLOADER == 1 or headphones.CONFIG.TORRENT_DOWNLOADER == 2:
         seed_snatched = myDB.action('SELECT * from snatched WHERE Status="Seed_Snatched" and AlbumID=?', [albumid]).fetchone()
         if seed_snatched:
             hash = seed_snatched['FolderName']
             torrent_removed = False
             logger.info(u'%s - %s. Checking if torrent has finished seeding and can be removed' % (release['ArtistName'], release['AlbumTitle']))
-            if headphones.TORRENT_DOWNLOADER == 1:
+            if headphones.CONFIG.TORRENT_DOWNLOADER == 1:
                 torrent_removed = transmission.removeTorrent(hash, True)
             else:
                 torrent_removed = utorrent.removeTorrent(hash, True)
@@ -455,89 +422,90 @@ def doPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list,
     pushmessage = release['ArtistName'] + ' - ' + release['AlbumTitle']
     statusmessage = "Download and Postprocessing completed"
 
-    if headphones.GROWL_ENABLED:
+    if headphones.CONFIG.GROWL_ENABLED:
         logger.info(u"Growl request")
         growl = notifiers.GROWL()
         growl.notify(pushmessage, statusmessage)
 
-    if headphones.PROWL_ENABLED:
+    if headphones.CONFIG.PROWL_ENABLED:
         logger.info(u"Prowl request")
         prowl = notifiers.PROWL()
         prowl.notify(pushmessage, statusmessage)
 
-    if headphones.XBMC_ENABLED:
+    if headphones.CONFIG.XBMC_ENABLED:
         xbmc = notifiers.XBMC()
-        if headphones.XBMC_UPDATE:
+        if headphones.CONFIG.XBMC_UPDATE:
             xbmc.update()
-        if headphones.XBMC_NOTIFY:
+        if headphones.CONFIG.XBMC_NOTIFY:
             xbmc.notify(release['ArtistName'],
                         release['AlbumTitle'],
                         album_art_path)
 
-    if headphones.LMS_ENABLED:
+    if headphones.CONFIG.LMS_ENABLED:
         lms = notifiers.LMS()
         lms.update()
 
-    if headphones.PLEX_ENABLED:
+    if headphones.CONFIG.PLEX_ENABLED:
         plex = notifiers.Plex()
-        if headphones.PLEX_UPDATE:
+        if headphones.CONFIG.PLEX_UPDATE:
             plex.update()
-        if headphones.PLEX_NOTIFY:
+        if headphones.CONFIG.PLEX_NOTIFY:
             plex.notify(release['ArtistName'],
                         release['AlbumTitle'],
                         album_art_path)
 
-    if headphones.NMA_ENABLED:
+    if headphones.CONFIG.NMA_ENABLED:
         nma = notifiers.NMA()
         nma.notify(release['ArtistName'], release['AlbumTitle'])
 
-    if headphones.PUSHALOT_ENABLED:
+    if headphones.CONFIG.PUSHALOT_ENABLED:
         logger.info(u"Pushalot request")
         pushalot = notifiers.PUSHALOT()
         pushalot.notify(pushmessage, statusmessage)
 
-    if headphones.SYNOINDEX_ENABLED:
+    if headphones.CONFIG.SYNOINDEX_ENABLED:
         syno = notifiers.Synoindex()
         for albumpath in albumpaths:
             syno.notify(albumpath)
 
-    if headphones.PUSHOVER_ENABLED:
+    if headphones.CONFIG.PUSHOVER_ENABLED:
         logger.info(u"Pushover request")
         pushover = notifiers.PUSHOVER()
         pushover.notify(pushmessage, "Headphones")
 
-    if headphones.PUSHBULLET_ENABLED:
+    if headphones.CONFIG.PUSHBULLET_ENABLED:
         logger.info(u"PushBullet request")
         pushbullet = notifiers.PUSHBULLET()
         pushbullet.notify(pushmessage, "Download and Postprocessing completed")
 
-    if headphones.TWITTER_ENABLED:
+    if headphones.CONFIG.TWITTER_ENABLED:
         logger.info(u"Sending Twitter notification")
         twitter = notifiers.TwitterNotifier()
         twitter.notify_download(pushmessage)
 
-    if headphones.OSX_NOTIFY_ENABLED:
+    if headphones.CONFIG.OSX_NOTIFY_ENABLED:
         logger.info(u"Sending OS X notification")
         osx_notify = notifiers.OSX_NOTIFY()
         osx_notify.notify(release['ArtistName'],
                           release['AlbumTitle'],
                           statusmessage)
 
-    if headphones.BOXCAR_ENABLED:
+    if headphones.CONFIG.BOXCAR_ENABLED:
         logger.info(u"Sending Boxcar2 notification")
         boxcar = notifiers.BOXCAR()
         boxcar.notify('Headphones processed: ' + pushmessage,
                       statusmessage, release['AlbumID'])
 
-    if headphones.SUBSONIC_ENABLED:
+    if headphones.CONFIG.SUBSONIC_ENABLED:
         logger.info(u"Sending Subsonic update")
         subsonic = notifiers.SubSonicNotifier()
         subsonic.notify(albumpaths)
 
-    if headphones.MPC_ENABLED:
+    if headphones.CONFIG.MPC_ENABLED:
         mpc = notifiers.MPC()
         mpc.notify()
 
+
 def embedAlbumArt(artwork, downloaded_track_list):
     logger.info('Embedding album art')
 
@@ -553,10 +521,11 @@ def embedAlbumArt(artwork, downloaded_track_list):
         try:
             f.art = artwork
             f.save()
-        except Exception, e:
+        except Exception as e:
             logger.error(u'Error embedding album art to: %s. Error: %s' % (downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), str(e)))
             continue
 
+
 def addAlbumArt(artwork, albumpath, release):
     logger.info('Adding album art to folder')
 
@@ -573,11 +542,11 @@ def addAlbumArt(artwork, albumpath, release):
                 '$year': year
                 }
 
-    album_art_name = helpers.replace_all(headphones.ALBUM_ART_FORMAT.strip(), values) + ".jpg"
+    album_art_name = helpers.replace_all(headphones.CONFIG.ALBUM_ART_FORMAT.strip(), values) + ".jpg"
 
     album_art_name = helpers.replace_illegal_chars(album_art_name).encode(headphones.SYS_ENCODING, 'replace')
 
-    if headphones.FILE_UNDERSCORES:
+    if headphones.CONFIG.FILE_UNDERSCORES:
         album_art_name = album_art_name.replace(' ', '_')
 
     if album_art_name.startswith('.'):
@@ -590,6 +559,7 @@ def addAlbumArt(artwork, albumpath, release):
         logger.error('Error saving album art: %s', e)
         return
 
+
 def cleanupFiles(albumpath):
     logger.info('Cleaning up files')
 
@@ -602,6 +572,7 @@ def cleanupFiles(albumpath):
                 except Exception as e:
                     logger.error(u'Could not remove file: %s. Error: %s' % (files.decode(headphones.SYS_ENCODING, 'replace'), e))
 
+
 def renameNFO(albumpath):
     logger.info('Renaming NFO')
 
@@ -615,15 +586,17 @@ def renameNFO(albumpath):
                 except Exception as e:
                     logger.error(u'Could not rename file: %s. Error: %s' % (os.path.join(r, file).decode(headphones.SYS_ENCODING, 'replace'), e))
 
+
 def moveFiles(albumpath, release, tracks):
+    logger.info("Moving files: %s" % albumpath)
     try:
         year = release['ReleaseDate'][:4]
     except TypeError:
-        year = ''
+        year = u''
 
     artist = release['ArtistName'].replace('/', '_')
     album = release['AlbumTitle'].replace('/', '_')
-    if headphones.FILE_UNDERSCORES:
+    if headphones.CONFIG.FILE_UNDERSCORES:
         artist = artist.replace(' ', '_')
         album = album.replace(' ', '_')
 
@@ -635,7 +608,7 @@ def moveFiles(albumpath, release, tracks):
         sortname = release['ArtistName']
 
     if sortname[0].isdigit():
-        firstchar = '0-9'
+        firstchar = u'0-9'
     else:
         firstchar = sortname[0]
 
@@ -661,7 +634,7 @@ def moveFiles(albumpath, release, tracks):
                 '$originalfolder': origfolder.lower()
             }
 
-    folder = helpers.replace_all(headphones.FOLDER_FORMAT.strip(), values, normalize=True)
+    folder = helpers.replace_all(headphones.CONFIG.FOLDER_FORMAT.strip(), values, normalize=True)
 
     folder = helpers.replace_illegal_chars(folder, type="folder")
     folder = folder.replace('./', '_/').replace('/.', '/_')
@@ -690,11 +663,11 @@ def moveFiles(albumpath, release, tracks):
     make_lossy_folder = False
     make_lossless_folder = False
 
-    lossy_destination_path = os.path.normpath(os.path.join(headphones.DESTINATION_DIR, folder)).encode(headphones.SYS_ENCODING, 'replace')
+    lossy_destination_path = os.path.normpath(os.path.join(headphones.CONFIG.DESTINATION_DIR, folder)).encode(headphones.SYS_ENCODING, 'replace')
-    lossless_destination_path = os.path.normpath(os.path.join(headphones.LOSSLESS_DESTINATION_DIR, folder)).encode(headphones.SYS_ENCODING, 'replace')
+    lossless_destination_path = os.path.normpath(os.path.join(headphones.CONFIG.LOSSLESS_DESTINATION_DIR, folder)).encode(headphones.SYS_ENCODING, 'replace')
 
     # If they set a destination dir for lossless media, only create the lossy folder if there is lossy media
-    if headphones.LOSSLESS_DESTINATION_DIR:
+    if headphones.CONFIG.LOSSLESS_DESTINATION_DIR:
         if lossy_media:
             make_lossy_folder = True
         if lossless_media:
@@ -703,7 +676,7 @@ def moveFiles(albumpath, release, tracks):
     else:
         make_lossy_folder = True
 
-    last_folder = headphones.FOLDER_FORMAT.strip().split('/')[-1]
+    last_folder = headphones.CONFIG.FOLDER_FORMAT.strip().split('/')[-1]
 
     if make_lossless_folder:
         # Only rename the folder if they use the album name, otherwise merge into existing folder
@@ -711,20 +684,20 @@ def moveFiles(albumpath, release, tracks):
 
             create_duplicate_folder = False
 
-            if headphones.REPLACE_EXISTING_FOLDERS:
+            if headphones.CONFIG.REPLACE_EXISTING_FOLDERS:
                 try:
                     shutil.rmtree(lossless_destination_path)
-                except Exception, e:
+                except Exception as e:
                     logger.error("Error deleting existing folder: %s. Creating duplicate folder. Error: %s" % (lossless_destination_path.decode(headphones.SYS_ENCODING, 'replace'), e))
                     create_duplicate_folder = True
 
-            if not headphones.REPLACE_EXISTING_FOLDERS or create_duplicate_folder:
+            if not headphones.CONFIG.REPLACE_EXISTING_FOLDERS or create_duplicate_folder:
                 temp_folder = folder
 
                 i = 1
                 while True:
                     newfolder = temp_folder + '[%i]' % i
-                    lossless_destination_path = os.path.normpath(os.path.join(headphones.LOSSLESS_DESTINATION_DIR, newfolder)).encode(headphones.SYS_ENCODING, 'replace')
+                    lossless_destination_path = os.path.normpath(os.path.join(headphones.CONFIG.LOSSLESS_DESTINATION_DIR, newfolder)).encode(headphones.SYS_ENCODING, 'replace')
                     if os.path.exists(lossless_destination_path):
                         i += 1
                     else:
@@ -734,7 +707,7 @@ def moveFiles(albumpath, release, tracks):
         if not os.path.exists(lossless_destination_path):
             try:
                 os.makedirs(lossless_destination_path)
-            except Exception, e:
+            except Exception as e:
                 logger.error('Could not create lossless folder for %s. (Error: %s)' % (release['AlbumTitle'], e))
                 if not make_lossy_folder:
                     return [albumpath]
@@ -744,20 +717,20 @@ def moveFiles(albumpath, release, tracks):
 
             create_duplicate_folder = False
 
-            if headphones.REPLACE_EXISTING_FOLDERS:
+            if headphones.CONFIG.REPLACE_EXISTING_FOLDERS:
                 try:
                     shutil.rmtree(lossy_destination_path)
-                except Exception, e:
+                except Exception as e:
                     logger.error("Error deleting existing folder: %s. Creating duplicate folder. Error: %s" % (lossy_destination_path.decode(headphones.SYS_ENCODING, 'replace'), e))
                     create_duplicate_folder = True
 
-            if not headphones.REPLACE_EXISTING_FOLDERS or create_duplicate_folder:
+            if not headphones.CONFIG.REPLACE_EXISTING_FOLDERS or create_duplicate_folder:
                 temp_folder = folder
 
                 i = 1
                 while True:
                     newfolder = temp_folder + '[%i]' % i
-                    lossy_destination_path = os.path.normpath(os.path.join(headphones.DESTINATION_DIR, newfolder)).encode(headphones.SYS_ENCODING, 'replace')
+                    lossy_destination_path = os.path.normpath(os.path.join(headphones.CONFIG.DESTINATION_DIR, newfolder)).encode(headphones.SYS_ENCODING, 'replace')
                     if os.path.exists(lossy_destination_path):
                         i += 1
                     else:
@@ -767,7 +740,7 @@ def moveFiles(albumpath, release, tracks):
         if not os.path.exists(lossy_destination_path):
             try:
                 os.makedirs(lossy_destination_path)
-            except Exception, e:
+            except Exception as e:
                 logger.error('Could not create folder for %s. Not moving: %s' % (release['AlbumTitle'], e))
                 return [albumpath]
 
@@ -795,7 +768,7 @@ def moveFiles(albumpath, release, tracks):
                 if moved_to_lossy_folder or moved_to_lossless_folder:
                     try:
                         os.remove(file_to_move)
-                    except Exception, e:
+                    except Exception as e:
                         logger.error("Error deleting file '" + file_to_move.decode(headphones.SYS_ENCODING, 'replace') + "' from source directory")
                 else:
                     logger.error("Error copying '" + file_to_move.decode(headphones.SYS_ENCODING, 'replace') + "'. Not deleting from download directory")
@@ -815,10 +788,10 @@ def moveFiles(albumpath, release, tracks):
     temp_fs = []
 
     if make_lossless_folder:
-        temp_fs.append(headphones.LOSSLESS_DESTINATION_DIR)
+        temp_fs.append(headphones.CONFIG.LOSSLESS_DESTINATION_DIR)
 
     if make_lossy_folder:
-        temp_fs.append(headphones.DESTINATION_DIR)
+        temp_fs.append(headphones.CONFIG.DESTINATION_DIR)
 
     for temp_f in temp_fs:
 
@@ -827,14 +800,14 @@ def moveFiles(albumpath, release, tracks):
             temp_f = os.path.join(temp_f, f)
 
             try:
-                os.chmod(os.path.normpath(temp_f).encode(headphones.SYS_ENCODING, 'replace'), int(headphones.FOLDER_PERMISSIONS, 8))
+                os.chmod(os.path.normpath(temp_f).encode(headphones.SYS_ENCODING, 'replace'), int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
-            except Exception, e:
+            except Exception as e:
                 logger.error("Error trying to change permissions on folder: %s. %s", temp_f, e)
 
     # If we failed to move all the files out of the directory, this will fail too
     try:
         shutil.rmtree(albumpath)
-    except Exception, e:
+    except Exception as e:
         logger.error('Could not remove directory: %s. %s', albumpath, e)
 
     destination_paths = []
@@ -846,6 +819,7 @@ def moveFiles(albumpath, release, tracks):
 
     return destination_paths
 
+
 def correctMetadata(albumid, release, downloaded_track_list):
 
     logger.info('Preparing to write metadata to tracks....')
@@ -863,8 +837,7 @@ def correctMetadata(albumid, release, downloaded_track_list):
                 lossy_items.append(beets.library.Item.from_path(downloaded_track))
             else:
                 logger.warn("Skipping: %s because it is not a mutagen friendly file format", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
-        except Exception, e:
+        except Exception as e:
-
             logger.error("Beets couldn't create an Item from: %s - not a media file? %s", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), str(e))
 
     for items in [lossy_items, lossless_items]:
@@ -874,7 +847,7 @@ def correctMetadata(albumid, release, downloaded_track_list):
 
         try:
             cur_artist, cur_album, candidates, rec = autotag.tag_album(items, search_artist=helpers.latinToAscii(release['ArtistName']), search_album=helpers.latinToAscii(release['AlbumTitle']))
-        except Exception, e:
+        except Exception as e:
             logger.error('Error getting recommendation: %s. Not writing metadata', e)
             return
         if str(rec) == 'recommendation.none':
@@ -897,40 +870,56 @@ def correctMetadata(albumid, release, downloaded_track_list):
             try:
                 item.write()
                 logger.info("Successfully applied metadata to: %s", item.path.decode(headphones.SYS_ENCODING, 'replace'))
-            except Exception, e:
+            except Exception as e:
                 logger.warn("Error writing metadata to '%s': %s", item.path.decode(headphones.SYS_ENCODING, 'replace'), str(e))
 
+
 def embedLyrics(downloaded_track_list):
     logger.info('Adding lyrics')
 
-    # TODO: If adding lyrics for flac & lossy, only fetch the lyrics once
+    # TODO: If adding lyrics for flac & lossy, only fetch the lyrics once and apply it to both files
-    # and apply it to both files
+    # TODO: Get beets to add automatically by enabling the plugin
+
+    lossy_items = []
+    lossless_items = []
+    lp = beetslyrics.LyricsPlugin()
+
     for downloaded_track in downloaded_track_list:
-        track_title = downloaded_track.decode(headphones.SYS_ENCODING, 'replace')
+
         try:
-            f = MediaFile(downloaded_track)
+            if any(downloaded_track.lower().endswith('.' + x.lower()) for x in headphones.LOSSLESS_MEDIA_FORMATS):
-        except:
+                lossless_items.append(beets.library.Item.from_path(downloaded_track))
-            logger.error('Could not read %s. Not checking lyrics', track_title)
+            elif any(downloaded_track.lower().endswith('.' + x.lower()) for x in headphones.LOSSY_MEDIA_FORMATS):
+                lossy_items.append(beets.library.Item.from_path(downloaded_track))
+            else:
+                logger.warn("Skipping: %s because it is not a mutagen friendly file format", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'))
+        except Exception as e:
+            logger.error("Beets couldn't create an Item from: %s - not a media file? %s", downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), str(e))
+
+    for items in [lossy_items, lossless_items]:
+
+        if not items:
             continue
 
-        if f.albumartist and f.title:
+        for item in items:
-            metalyrics = lyrics.getLyrics(f.albumartist, f.title)
-        elif f.artist and f.title:
-            metalyrics = lyrics.getLyrics(f.artist, f.title)
-        else:
-            logger.info('No artist/track metadata found for track: %s. Not fetching lyrics', track_title)
-            metalyrics = None
 
-        if metalyrics:
+            lyrics = None
-            logger.debug('Adding lyrics to: %s', track_title)
+            for artist, titles in beetslyrics.search_pairs(item):
-            f.lyrics = metalyrics
+                lyrics = [lp.get_lyrics(artist, title) for title in titles]
+                if any(lyrics):
+                    break
+
+            lyrics = u"\n\n---\n\n".join([l for l in lyrics if l])
+
+            if lyrics:
+                logger.debug('Adding lyrics to: %s', item.title)
+                item.lyrics = lyrics
                 try:
-                f.save()
+                    item.write()
-            except:
+                except Exception as e:
-                logger.error('Cannot save lyrics to: %s. Skipping', track_title)
+                    logger.error('Cannot save lyrics to: %s. Skipping', item.title)
-                continue
             else:
-            logger.debug('No lyrics found for track: %s', track_title)
+                logger.debug('No lyrics found for track: %s', item.title)
 
 
 def renameFiles(albumpath, downloaded_track_list, release):
@@ -997,12 +986,11 @@ def renameFiles(albumpath, downloaded_track_list, release):
 
             ext = os.path.splitext(downloaded_track)[1]
 
-            new_file_name = helpers.replace_all(headphones.FILE_FORMAT.strip(), values).replace('/','_') + ext
+            new_file_name = helpers.replace_all(headphones.CONFIG.FILE_FORMAT.strip(), values).replace('/', '_') + ext
-
 
         new_file_name = helpers.replace_illegal_chars(new_file_name).encode(headphones.SYS_ENCODING, 'replace')
 
-        if headphones.FILE_UNDERSCORES:
+        if headphones.CONFIG.FILE_UNDERSCORES:
             new_file_name = new_file_name.replace(' ', '_')
 
         if new_file_name.startswith('.'):
@@ -1017,10 +1005,11 @@ def renameFiles(albumpath, downloaded_track_list, release):
         logger.debug('Renaming %s ---> %s', downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), new_file_name.decode(headphones.SYS_ENCODING, 'replace'))
         try:
             os.rename(downloaded_track, new_file)
-        except Exception, e:
+        except Exception as e:
             logger.error('Error renaming file: %s. Error: %s', downloaded_track.decode(headphones.SYS_ENCODING, 'replace'), e)
             continue
 
+
 def updateFilePermissions(albumpaths):
 
     for folder in albumpaths:
@@ -1029,11 +1018,12 @@ def updateFilePermissions(albumpaths):
             for files in f:
                 full_path = os.path.join(r, files)
                 try:
-                    os.chmod(full_path, int(headphones.FILE_PERMISSIONS, 8))
+                    os.chmod(full_path, int(headphones.CONFIG.FILE_PERMISSIONS, 8))
                 except:
                     logger.error("Could not change permissions for file: %s", full_path)
                     continue
 
+
 def renameUnprocessedFolder(albumpath):
 
     i = 0
@@ -1050,6 +1040,7 @@ def renameUnprocessedFolder(albumpath):
             os.rename(albumpath, new_folder_name)
             return
 
+
 def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
 
     if album_dir:
@@ -1059,15 +1050,15 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
         download_dirs = []
         if dir:
             download_dirs.append(dir.encode(headphones.SYS_ENCODING, 'replace'))
-        if headphones.DOWNLOAD_DIR and not dir:
+        if headphones.CONFIG.DOWNLOAD_DIR and not dir:
-            download_dirs.append(headphones.DOWNLOAD_DIR.encode(headphones.SYS_ENCODING, 'replace'))
+            download_dirs.append(headphones.CONFIG.DOWNLOAD_DIR.encode(headphones.SYS_ENCODING, 'replace'))
-        if headphones.DOWNLOAD_TORRENT_DIR and not dir:
+        if headphones.CONFIG.DOWNLOAD_TORRENT_DIR and not dir:
-            download_dirs.append(headphones.DOWNLOAD_TORRENT_DIR.encode(headphones.SYS_ENCODING, 'replace'))
+            download_dirs.append(headphones.CONFIG.DOWNLOAD_TORRENT_DIR.encode(headphones.SYS_ENCODING, 'replace'))
 
         # If DOWNLOAD_DIR and DOWNLOAD_TORRENT_DIR are the same, remove the duplicate to prevent us from trying to process the same folder twice.
         download_dirs = list(set(download_dirs))
+        logger.debug('Post processing folders: %s', download_dirs)
 
-        logger.info('Checking to see if there are any folders to process in download_dir(s): %s', download_dirs)
         # Get a list of folders in the download_dir
         folders = []
 
@@ -1086,10 +1077,13 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                     else:
                         folders.append(path_to_folder)
 
-        if len(folders):
+    # Log number of folders
-            logger.info('Found %i folders to process', len(folders))
+    if folders:
+        logger.info('Found %i folders to process.', len(folders))
+        logger.debug('Expanded post processing folders: %s', folders)
     else:
-            logger.info('Found no folders to process in: %s', download_dirs)
+        logger.info('Found no folders to process. Aborting.')
+        return
 
     # Parse the folder names to get artist album info
     myDB = db.DBConnection()
@@ -1107,7 +1101,7 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
         snatched = myDB.action('SELECT AlbumID, Title, Kind, Status from snatched WHERE FolderName LIKE ?', [folder_basename]).fetchone()
 
         if snatched:
-            if headphones.KEEP_TORRENT_FILES and snatched['Kind'] == 'torrent' and snatched['Status'] == 'Processed':
+            if headphones.CONFIG.KEEP_TORRENT_FILES and snatched['Kind'] == 'torrent' and snatched['Status'] == 'Processed':
                 logger.info('%s is a torrent folder being preserved for seeding and has already been processed. Skipping.', folder_basename)
                 continue
             else:
@@ -1115,68 +1109,13 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                 verify(snatched['AlbumID'], folder, snatched['Kind'])
                 continue
 
-        # Attempt 2a: parse the folder name into a valid format
+        # Attempt 2: strip release group id from filename
-        try:
-            logger.debug('Attempting to extract name, album and year from folder name')
-            name, album, year = helpers.extract_data(folder_basename)
-        except Exception as e:
-            name = album = year = None
-
-        if name and album:
-            release = myDB.action('SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE ArtistName LIKE ? and AlbumTitle LIKE ?', [name, album]).fetchone()
-            if release:
-                logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
-                verify(release['AlbumID'], folder)
-                continue
-            else:
-                logger.info('Querying MusicBrainz for the release group id for: %s - %s', name, album)
-                try:
-                    rgid = mb.findAlbumID(helpers.latinToAscii(name), helpers.latinToAscii(album))
-                except:
-                    logger.error('Can not get release information for this album')
-                    rgid = None
-
-                if rgid:
-                    verify(rgid, folder)
-                    continue
-                else:
-                    logger.info('No match found on MusicBrainz for: %s - %s', name, album)
-
-        # Attempt 2b: deduce meta data into a valid format
-        try:
-            logger.debug('Attempting to extract name, album and year from metadata')
-            name, album, year = helpers.extract_metadata(folder)
-        except Exception as e:
-            name = album = year = None
-
-        if name and album:
-            release = myDB.action('SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE ArtistName LIKE ? and AlbumTitle LIKE ?', [name, album]).fetchone()
-            if release:
-                logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
-                verify(release['AlbumID'], folder)
-                continue
-            else:
-                logger.info('Querying MusicBrainz for the release group id for: %s - %s', name, album)
-                try:
-                    rgid = mb.findAlbumID(helpers.latinToAscii(name), helpers.latinToAscii(album))
-                except:
-                    logger.error('Can not get release information for this album')
-                    rgid = None
-
-                if rgid:
-                    verify(rgid, folder)
-                    continue
-                else:
-                    logger.info('No match found on MusicBrainz for: %s - %s', name, album)
-
-        # Attempt 3: strip release group id from filename
-        try:
         logger.debug('Attempting to extract release group from folder name')
+
+        try:
             possible_rgid = folder_basename[-36:]
-            # re pattern match: [0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}
             rgid = uuid.UUID(possible_rgid)
         except:
-            logger.info("Couldn't parse '%s' into any valid format. If adding albums from another source, they must be in an 'Artist - Album [Year]' format, or end with the musicbrainz release group id", folder_basename)
             rgid = possible_rgid = None
 
         if rgid:
@@ -1187,11 +1126,75 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                 verify(release['AlbumID'], folder, forced=True)
                 continue
             else:
-                logger.info('Found a (possibly) valid Musicbrainz identifier in album folder name - continuing post-processing')
+                logger.info('Found a (possibly) valid Musicbrainz realse group id in album folder name.')
                 verify(rgid, folder, forced=True)
                 continue
 
-        # Attempt 4: Hail mary. Just assume the folder name is the album name if it doesn't have a separator in it
+        # Attempt 3a: parse the folder name into a valid format
+        logger.debug('Attempting to extract name, album and year from folder name')
+
+        try:
+            name, album, year = helpers.extract_data(folder_basename)
+        except Exception:
+            name = album = year = None
+
+        if name and album:
+            release = myDB.action('SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE ArtistName LIKE ? and AlbumTitle LIKE ?', [name, album]).fetchone()
+            if release:
+                logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
+                verify(release['AlbumID'], folder)
+                continue
+            else:
+                logger.info('Querying MusicBrainz for the release group id for: %s - %s', name, album)
+                try:
+                    rgid = mb.findAlbumID(helpers.latinToAscii(name), helpers.latinToAscii(album))
+                except:
+                    logger.error('Can not get release information for this album')
+                    rgid = None
+
+                if rgid:
+                    verify(rgid, folder)
+                    continue
+                else:
+                    logger.info('No match found on MusicBrainz for: %s - %s', name, album)
+
+        # Attempt 3b: deduce meta data into a valid format
+        logger.debug('Attempting to extract name, album and year from metadata')
+
+        try:
+            name, album, year = helpers.extract_metadata(folder)
+        except Exception:
+            name = album = None
+
+        # Check if there's a cue to split
+        if headphones.CONFIG.CUE_SPLIT and not name and not album and helpers.cue_split(folder):
+            try:
+                name, album, year = helpers.extract_metadata(folder)
+            except Exception:
+                name = album = None
+
+        if name and album:
+            release = myDB.action('SELECT AlbumID, ArtistName, AlbumTitle from albums WHERE ArtistName LIKE ? and AlbumTitle LIKE ?', [name, album]).fetchone()
+            if release:
+                logger.info('Found a match in the database: %s - %s. Verifying to make sure it is the correct album', release['ArtistName'], release['AlbumTitle'])
+                verify(release['AlbumID'], folder)
+                continue
+            else:
+                logger.info('Querying MusicBrainz for the release group id for: %s - %s', name, album)
+                try:
+                    rgid = mb.findAlbumID(helpers.latinToAscii(name), helpers.latinToAscii(album))
+                except:
+                    logger.error('Can not get release information for this album')
+                    rgid = None
+
+                if rgid:
+                    verify(rgid, folder)
+                    continue
+                else:
+                    logger.info('No match found on MusicBrainz for: %s - %s', name, album)
+
+        # Attempt 4: Hail mary. Just assume the folder name is the album name
+        # if it doesn't have a separator in it
         logger.debug('Attempt to extract album name by assuming it is the folder name')
 
         if '-' not in folder_basename:
@@ -1213,3 +1216,9 @@ def forcePostProcess(dir=None, expand_subfolders=True, album_dir=None):
                     continue
                 else:
                     logger.info('No match found on MusicBrainz for: %s - %s', name, album)
+
+        # Fail here
+        logger.info("Couldn't parse '%s' into any valid format. If adding " \
+            "albums from another source, they must be in an 'Artist - Album " \
+            "[Year]' format, or end with the musicbrainz release group id.",
+            folder_basename)

headphones/request.py

@@ -18,16 +18,28 @@ from headphones import logger
 from xml.dom import minidom
 from bs4 import BeautifulSoup
 
+import time
 import requests
 import feedparser
 import headphones
+import collections
+
+# Dictionary with last request times, for rate limiting.
+last_requests = collections.defaultdict(int)
+
 
 def request_response(url, method="get", auto_raise=True,
-    whitelist_status_code=None, **kwargs):
+    whitelist_status_code=None, rate_limit=None, **kwargs):
     """
     Convenient wrapper for `requests.get', which will capture the exceptions and
     log them. On success, the Response object is returned. In case of a
     exception, None is returned.
+
+    Additionally, there is support for rate limiting. To use this feature,
+    supply a tuple of (lock, request_limit). The lock is used to make sure no
+    other request with the same lock is executed. The request limit is the
+    minimal time between two requests (and so 1/request_limit is the number of
+    requests per seconds).
     """
 
     # Convert whitelist_status_code to a list if needed
@@ -36,14 +48,40 @@ def request_response(url, method="get", auto_raise=True,
 
     # Disable verification of SSL certificates if requested. Note: this could
     # pose a security issue!
-    kwargs["verify"] = headphones.VERIFY_SSL_CERT
+    kwargs["verify"] = bool(headphones.CONFIG.VERIFY_SSL_CERT)
 
     # Map method to the request.XXX method. This is a simple hack, but it allows
     # requests to apply more magic per method. See lib/requests/api.py.
     request_method = getattr(requests, method.lower())
 
+    # Enfore request rate limit if applicable. This uses the lock so there
+    # is synchronized access to the API. When N threads enter this method, the
+    # first will pass trough, since there there was no last request recorded.
+    # The last request time will be set. Then, the second thread will unlock,
+    # and see that the last request was X seconds ago. It will sleep
+    # (request_limit - X) seconds, and then continue. Then the third one will
+    # unblock, and so on. After all threads finished, the total time will at
+    # least be (N * request_limit) seconds. If some request takes longer than
+    # request_limit seconds, the next unblocked thread will wait less.
+    if rate_limit:
+        lock, request_limit = rate_limit
+
+        with lock:
+            delta = time.time() - last_requests[lock]
+            limit = int(1.0 / request_limit)
+
+            if delta < request_limit:
+                logger.debug("Sleeping %.2f seconds for request, limit " \
+                    "is %d req/sec.", request_limit - delta, limit)
+
+                # Sleep the remaining time
+                time.sleep(request_limit - delta)
+
+            # Update last request time.
+            last_requests[lock] = time.time()
+
     try:
-        # Request the URL
+        # Request URL and wait for response
         logger.debug("Requesting URL via %s method: %s", method.upper(), url)
         response = request_method(url, **kwargs)
 
@@ -88,6 +126,7 @@ def request_response(url, method="get", auto_raise=True,
     except requests.RequestException as e:
         logger.error("Request raised exception: %s", e)
 
+
 def request_soup(url, **kwargs):
     """
     Wrapper for `request_response', which will return a BeatifulSoup object if
@@ -100,6 +139,7 @@ def request_soup(url, **kwargs):
     if response is not None:
         return BeautifulSoup(response.content, parser)
 
+
 def request_minidom(url, **kwargs):
     """
     Wrapper for `request_response', which will return a Minidom object if no
@@ -111,6 +151,7 @@ def request_minidom(url, **kwargs):
     if response is not None:
         return minidom.parseString(response.content)
 
+
 def request_json(url, **kwargs):
     """
     Wrapper for `request_response', which will decode the response as JSON
@@ -138,6 +179,7 @@ def request_json(url, **kwargs):
             if headphones.VERBOSE:
                 server_message(response)
 
+
 def request_content(url, **kwargs):
     """
     Wrapper for `request_response', which will return the raw content.
@@ -148,6 +190,7 @@ def request_content(url, **kwargs):
     if response is not None:
         return response.content
 
+
 def request_feed(url, **kwargs):
     """
     Wrapper for `request_response', which will return a feed object.
@@ -158,6 +201,7 @@ def request_feed(url, **kwargs):
     if response is not None:
         return feedparser.parse(response.content)
 
+
 def server_message(response):
     """
     Extract server message from response and log in to logger with DEBUG level.

headphones/sab.py

@@ -19,7 +19,6 @@
 
 import MultipartPostHandler
 import headphones
-import datetime
 import cookielib
 import urllib2
 import httplib
@@ -28,20 +27,21 @@ import ast
 
 from headphones.common import USER_AGENT
 from headphones import logger
-from headphones import notifiers, helpers
+from headphones import helpers
+
 
 def sendNZB(nzb):
 
     params = {}
 
-    if headphones.SAB_USERNAME:
+    if headphones.CONFIG.SAB_USERNAME:
-        params['ma_username'] = headphones.SAB_USERNAME
+        params['ma_username'] = headphones.CONFIG.SAB_USERNAME
-    if headphones.SAB_PASSWORD:
+    if headphones.CONFIG.SAB_PASSWORD:
-        params['ma_password'] = headphones.SAB_PASSWORD
+        params['ma_password'] = headphones.CONFIG.SAB_PASSWORD
-    if headphones.SAB_APIKEY:
+    if headphones.CONFIG.SAB_APIKEY:
-        params['apikey'] = headphones.SAB_APIKEY
+        params['apikey'] = headphones.CONFIG.SAB_APIKEY
-    if headphones.SAB_CATEGORY:
+    if headphones.CONFIG.SAB_CATEGORY:
-        params['cat'] = headphones.SAB_CATEGORY
+        params['cat'] = headphones.CONFIG.SAB_CATEGORY
 
     # if it's a normal result we just pass SAB the URL
     if nzb.resultType == "nzb":
@@ -64,13 +64,13 @@ def sendNZB(nzb):
         params['mode'] = 'addfile'
         multiPartParams = {"nzbfile": (helpers.latinToAscii(nzb.name) + ".nzb", nzbdata)}
 
-    if not headphones.SAB_HOST.startswith('http'):
+    if not headphones.CONFIG.SAB_HOST.startswith('http'):
-        headphones.SAB_HOST = 'http://' + headphones.SAB_HOST
+        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
 
-    if headphones.SAB_HOST.endswith('/'):
+    if headphones.CONFIG.SAB_HOST.endswith('/'):
-        headphones.SAB_HOST = headphones.SAB_HOST[0:len(headphones.SAB_HOST)-1]
+        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[0:len(headphones.CONFIG.SAB_HOST) - 1]
 
-    url = headphones.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
+    url = headphones.CONFIG.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
 
     try:
 
@@ -87,25 +87,25 @@ def sendNZB(nzb):
 
             f = opener.open(req)
 
-    except (EOFError, IOError), e:
+    except (EOFError, IOError) as e:
         logger.error(u"Unable to connect to SAB with URL: %s" % url)
         return False
 
-    except httplib.InvalidURL, e:
+    except httplib.InvalidURL as e:
-        logger.error(u"Invalid SAB host, check your config. Current host: %s" % headphones.SAB_HOST)
+        logger.error(u"Invalid SAB host, check your config. Current host: %s" % headphones.CONFIG.SAB_HOST)
         return False
 
-    except Exception, e:
+    except Exception as e:
         logger.error(u"Error: " + str(e))
         return False
 
-    if f == None:
+    if f is None:
         logger.info(u"No data returned from SABnzbd, NZB not sent")
         return False
 
     try:
         result = f.readlines()
-    except Exception, e:
+    except Exception as e:
         logger.info(u"Error trying to get result from SAB, NZB not sent: ")
         return False
 
@@ -127,30 +127,31 @@ def sendNZB(nzb):
         logger.info(u"Unknown failure sending NZB to sab. Return text is: " + sabText)
         return False
 
+
 def checkConfig():
 
     params = {'mode': 'get_config',
                'section': 'misc'
                }
 
-    if headphones.SAB_USERNAME:
+    if headphones.CONFIG.SAB_USERNAME:
-        params['ma_username'] = headphones.SAB_USERNAME
+        params['ma_username'] = headphones.CONFIG.SAB_USERNAME
-    if headphones.SAB_PASSWORD:
+    if headphones.CONFIG.SAB_PASSWORD:
-        params['ma_password'] = headphones.SAB_PASSWORD
+        params['ma_password'] = headphones.CONFIG.SAB_PASSWORD
-    if headphones.SAB_APIKEY:
+    if headphones.CONFIG.SAB_APIKEY:
-        params['apikey'] = headphones.SAB_APIKEY
+        params['apikey'] = headphones.CONFIG.SAB_APIKEY
 
-    if not headphones.SAB_HOST.startswith('http'):
+    if not headphones.CONFIG.SAB_HOST.startswith('http'):
-        headphones.SAB_HOST = 'http://' + headphones.SAB_HOST
+        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
 
-    if headphones.SAB_HOST.endswith('/'):
+    if headphones.CONFIG.SAB_HOST.endswith('/'):
-        headphones.SAB_HOST = headphones.SAB_HOST[0:len(headphones.SAB_HOST)-1]
+        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[0:len(headphones.CONFIG.SAB_HOST) - 1]
 
-    url = headphones.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
+    url = headphones.CONFIG.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
 
     try:
         f = urllib.urlopen(url).read()
-    except Exception, e:
+    except Exception:
         logger.warn("Unable to read SABnzbd config file - cannot determine renaming options (might affect auto & forced post processing)")
         return (0, 0)
 

headphones/searcher_rutracker.py

@@ -4,8 +4,6 @@
 # Headphones rutracker.org search
 # Functions called from searcher.py
 
-from headphones import logger, db, utorrent
-
 from bencode import bencode as bencode, bdecode
 from urlparse import urlparse
 from bs4 import BeautifulSoup
@@ -20,6 +18,9 @@ import urllib
 import re
 import os
 
+from headphones import db, logger
+
+
 class Rutracker():
 
     logged_in = False
@@ -197,7 +198,7 @@ class Rutracker():
                             decoded = bdecode(torrent)
                             metainfo = decoded['info']
                         page.close()
-                    except Exception, e:
+                    except Exception as e:
                         logger.error('Error getting torrent: %s' % e)
                         return False
 
@@ -294,7 +295,7 @@ class Rutracker():
             os.umask(prev)
 
             # Add file to utorrent
-            if headphones.TORRENT_DOWNLOADER == 2:
+            if headphones.CONFIG.TORRENT_DOWNLOADER == 2:
                 self.utorrent_add_file(download_path)
 
         except Exception as e:
@@ -306,7 +307,7 @@ class Rutracker():
     #TODO get this working in utorrent.py
     def utorrent_add_file(self, filename):
 
-        host = headphones.UTORRENT_HOST
+        host = headphones.CONFIG.UTORRENT_HOST
         if not host.startswith('http'):
             host = 'http://' + host
         if host.endswith('/'):
@@ -315,8 +316,8 @@ class Rutracker():
             host = host[:-4]
 
         base_url = host
-        username = headphones.UTORRENT_USERNAME
+        username = headphones.CONFIG.UTORRENT_USERNAME
-        password = headphones.UTORRENT_PASSWORD
+        password = headphones.CONFIG.UTORRENT_PASSWORD
 
         session = requests.Session()
         url = base_url + '/gui/'
@@ -346,4 +347,3 @@ class Rutracker():
             except Exception:
                 logger.exception('Error adding file to utorrent')
                 return
-

headphones/torrentfinished.py

@@ -20,6 +20,8 @@ from headphones import db, utorrent, transmission, logger
 postprocessor_lock = threading.Lock()
 
 # Remove Torrent + data if Post Processed and finished Seeding
+
+
 def checkTorrentFinished():
 
     logger.info("Checking if any torrents have finished seeding and can be removed")
@@ -33,7 +35,7 @@ def checkTorrentFinished():
             hash = album['FolderName']
             albumid = album['AlbumID']
             torrent_removed = False
-            if headphones.TORRENT_DOWNLOADER == 1:
+            if headphones.CONFIG.TORRENT_DOWNLOADER == 1:
                 torrent_removed = transmission.removeTorrent(hash, True)
             else:
                 torrent_removed = utorrent.removeTorrent(hash, True)

headphones/transmission.py

@@ -13,9 +13,8 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger, notifiers, request
+from headphones import logger, request
 
-import re
 import time
 import json
 import base64
@@ -27,15 +26,16 @@ import headphones
 # TODO: Store the session id so we don't need to make 2 calls
 #       Store torrent id so we can check up on it
 
+
 def addTorrent(link):
     method = 'torrent-add'
 
     if link.endswith('.torrent'):
         with open(link, 'rb') as f:
             metainfo = str(base64.b64encode(f.read()))
-        arguments = {'metainfo': metainfo, 'download-dir':headphones.DOWNLOAD_TORRENT_DIR}
+        arguments = {'metainfo': metainfo, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}
     else:
-        arguments = {'filename': link, 'download-dir': headphones.DOWNLOAD_TORRENT_DIR}
+        arguments = {'filename': link, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}
 
     response = torrentAction(method, arguments)
 
@@ -44,13 +44,10 @@ def addTorrent(link):
 
     if response['result'] == 'success':
         if 'torrent-added' in response['arguments']:
-            name = response['arguments']['torrent-added']['name']
             retid = response['arguments']['torrent-added']['hashString']
         elif 'torrent-duplicate' in response['arguments']:
-            name = response['arguments']['torrent-duplicate']['name']
             retid = response['arguments']['torrent-duplicate']['hashString']
         else:
-            name = link
             retid = False
 
         logger.info(u"Torrent sent to Transmission successfully")
@@ -60,6 +57,7 @@ def addTorrent(link):
         logger.info('Transmission returned status %s' % response['result'])
         return False
 
+
 def getTorrentFolder(torrentid):
     method = 'torrent-get'
     arguments = {'ids': torrentid, 'fields': ['name', 'percentDone']}
@@ -80,6 +78,7 @@ def getTorrentFolder(torrentid):
 
     return torrent_folder_name
 
+
 def setSeedRatio(torrentid, ratio):
     method = 'torrent-set'
     if ratio != 0:
@@ -91,6 +90,7 @@ def setSeedRatio(torrentid, ratio):
     if not response:
         return False
 
+
 def removeTorrent(torrentid, remove_data=False):
 
     method = 'torrent-get'
@@ -120,11 +120,12 @@ def removeTorrent(torrentid, remove_data = False):
 
     return False
 
+
 def torrentAction(method, arguments):
 
-    host = headphones.TRANSMISSION_HOST
+    host = headphones.CONFIG.TRANSMISSION_HOST
-    username = headphones.TRANSMISSION_USERNAME
+    username = headphones.CONFIG.TRANSMISSION_USERNAME
-    password = headphones.TRANSMISSION_PASSWORD
+    password = headphones.CONFIG.TRANSMISSION_PASSWORD
     sessionid = None
 
     if not host.startswith('http'):
@@ -142,11 +143,13 @@ def torrentAction(method, arguments):
         i = host.rfind(':')
         if i >= 0:
             possible_port = host[i + 1:]
+            host = host + "/rpc"
             try:
                 port = int(possible_port)
+                if port:
                     host = host + "/transmission/rpc"
             except ValueError:
-                host = host + "/rpc"
+                logger.debug('No port, assuming not transmission')
         else:
             logger.error('Transmission port missing')
             return

headphones/updater.py

@@ -13,10 +13,9 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import headphones
-
 from headphones import logger, db, importer
 
+
 def dbUpdate(forcefull=False):
 
     myDB = db.DBConnection()

headphones/utorrent.py

@@ -13,14 +13,21 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import urllib, urllib2, urlparse, cookielib
+import urllib
-import json, re, os, time
+import urllib2
+import urlparse
+import cookielib
+import json
+import re
+import os
+import time
 
 import headphones
 
 from headphones import logger
 from collections import namedtuple
 
+
 class utorrentclient(object):
 
     TOKEN_REGEX = "<div id='token' style='display:none;'>([^<>]+)</div>"
@@ -28,7 +35,7 @@ class utorrentclient(object):
 
     def __init__(self, base_url=None, username=None, password=None,):
 
-        host = headphones.UTORRENT_HOST
+        host = headphones.CONFIG.UTORRENT_HOST
         if not host.startswith('http'):
             host = 'http://' + host
 
@@ -39,8 +46,8 @@ class utorrentclient(object):
             host = host[:-4]
 
         self.base_url = host
-        self.username = headphones.UTORRENT_USERNAME
+        self.username = headphones.CONFIG.UTORRENT_USERNAME
-        self.password = headphones.UTORRENT_PASSWORD
+        self.password = headphones.CONFIG.UTORRENT_PASSWORD
         self.opener = self._make_opener('uTorrent', self.base_url, self.username, self.password)
         self.token = self._get_token()
         #TODO refresh token, when necessary
@@ -156,12 +163,14 @@ class utorrentclient(object):
             logger.debug('URL: ' + str(url))
             logger.debug('uTorrent webUI raised the following error: ' + str(err))
 
+
 def labelTorrent(hash):
-    label = headphones.UTORRENT_LABEL
+    label = headphones.CONFIG.UTORRENT_LABEL
     uTorrentClient = utorrentclient()
     if label:
         uTorrentClient.setprops(hash, 'label', label)
 
+
 def removeTorrent(hash, remove_data=False):
     uTorrentClient = utorrentclient()
     status, torrentList = uTorrentClient.list()
@@ -177,6 +186,7 @@ def removeTorrent(hash, remove_data = False):
                 return False
     return False
 
+
 def setSeedRatio(hash, ratio):
     uTorrentClient = utorrentclient()
     uTorrentClient.setprops(hash, 'seed_override', '1')
@@ -186,6 +196,7 @@ def setSeedRatio(hash, ratio):
         # TODO passing -1 should be unlimited
         uTorrentClient.setprops(hash, 'seed_ratio', -10)
 
+
 def dirTorrent(hash, cacheid=None, return_name=None):
 
     uTorrentClient = utorrentclient()
@@ -212,6 +223,7 @@ def dirTorrent(hash, cacheid=None, return_name=None):
 
     return None, None
 
+
 def addTorrent(link, hash):
     uTorrentClient = utorrentclient()
 
@@ -230,7 +242,7 @@ def addTorrent(link, hash):
     # If there's no folder yet then it's probably a magnet, try until folder is populated
     if torrent_folder == active_dir or not torrent_folder:
         tries = 1
-        while (torrent_folder == active_dir or torrent_folder == None) and tries <= 10:
+        while (torrent_folder == active_dir or torrent_folder is None) and tries <= 10:
             tries += 1
             time.sleep(6)
             torrent_folder, cacheid = dirTorrent(hash, cacheid)
@@ -241,8 +253,11 @@ def addTorrent(link, hash):
         return torrent_folder
     else:
         labelTorrent(hash)
+        if headphones.SYS_PLATFORM != "win32":
+            torrent_folder = torrent_folder.replace('\\', '/')
         return os.path.basename(os.path.normpath(torrent_folder))
 
+
 def getSettingsDirectories():
     uTorrentClient = utorrentclient()
     settings = uTorrentClient.get_settings()
@@ -253,4 +268,3 @@ def getSettingsDirectories():
     if 'dir_completed_download' in settings:
         completed = settings['dir_completed_download'][2]
     return active, completed
-

headphones/versioncheck.py

@@ -22,10 +22,11 @@ import subprocess
 
 from headphones import logger, version, request
 
+
 def runGit(args):
 
-    if headphones.GIT_PATH:
+    if headphones.CONFIG.GIT_PATH:
-        git_locations = ['"'+headphones.GIT_PATH+'"']
+        git_locations = ['"' + headphones.CONFIG.GIT_PATH + '"']
     else:
         git_locations = ['git']
 
@@ -59,6 +60,7 @@ def runGit(args):
 
     return (output, err)
 
+
 def getVersion():
 
     if version.HEADPHONES_VERSION.startswith('win32build'):
@@ -82,16 +84,16 @@ def getVersion():
             logger.error('Output doesn\'t look like a hash, not using it')
             cur_commit_hash = None
 
-        if headphones.DO_NOT_OVERRIDE_GIT_BRANCH and headphones.GIT_BRANCH:
+        if headphones.CONFIG.DO_NOT_OVERRIDE_GIT_BRANCH and headphones.CONFIG.GIT_BRANCH:
-            branch_name = headphones.GIT_BRANCH
+            branch_name = headphones.CONFIG.GIT_BRANCH
 
         else:
             branch_name, err = runGit('rev-parse --abbrev-ref HEAD')
             branch_name = branch_name
 
-            if not branch_name and headphones.GIT_BRANCH:
+            if not branch_name and headphones.CONFIG.GIT_BRANCH:
-                logger.error('Could not retrieve branch name from git. Falling back to %s' % headphones.GIT_BRANCH)
+                logger.error('Could not retrieve branch name from git. Falling back to %s' % headphones.CONFIG.GIT_BRANCH)
-                branch_name = headphones.GIT_BRANCH
+                branch_name = headphones.CONFIG.GIT_BRANCH
             if not branch_name:
                 logger.error('Could not retrieve branch name from git. Defaulting to master')
                 branch_name = 'master'
@@ -111,16 +113,17 @@ def getVersion():
             current_version = f.read().strip(' \n\r')
 
         if current_version:
-            return current_version, headphones.GIT_BRANCH
+            return current_version, headphones.CONFIG.GIT_BRANCH
         else:
             return None, 'master'
 
+
 def checkGithub():
     headphones.COMMITS_BEHIND = 0
 
     # Get the latest version available from github
     logger.info('Retrieving latest version information from GitHub')
-    url = 'https://api.github.com/repos/%s/headphones/commits/%s' % (headphones.GIT_USER, headphones.GIT_BRANCH)
+    url = 'https://api.github.com/repos/%s/headphones/commits/%s' % (headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
     version = request.request_json(url, timeout=20, validator=lambda x: type(x) == dict)
 
     if version is None:
@@ -140,7 +143,7 @@ def checkGithub():
         return headphones.LATEST_VERSION
 
     logger.info('Comparing currently installed version with latest GitHub version')
-    url = 'https://api.github.com/repos/%s/headphones/compare/%s...%s' % (headphones.GIT_USER, headphones.LATEST_VERSION, headphones.CURRENT_VERSION)
+    url = 'https://api.github.com/repos/%s/headphones/compare/%s...%s' % (headphones.CONFIG.GIT_USER, headphones.LATEST_VERSION, headphones.CURRENT_VERSION)
     commits = request.request_json(url, timeout=20, whitelist_status_code=404, validator=lambda x: type(x) == dict)
 
     if commits is None:
@@ -161,12 +164,13 @@ def checkGithub():
 
     return headphones.LATEST_VERSION
 
+
 def update():
     if headphones.INSTALL_TYPE == 'win':
         logger.info('Windows .exe updating not supported yet.')
 
     elif headphones.INSTALL_TYPE == 'git':
-        output, err = runGit('pull origin ' + headphones.GIT_BRANCH)
+        output, err = runGit('pull origin ' + headphones.CONFIG.GIT_BRANCH)
 
         if not output:
             logger.error('Couldn\'t download latest version')
@@ -181,7 +185,7 @@ def update():
                 logger.info('Output: ' + str(output))
 
     else:
-        tar_download_url = 'https://github.com/%s/headphones/tarball/%s' % (headphones.GIT_USER, headphones.GIT_BRANCH)
+        tar_download_url = 'https://github.com/%s/headphones/tarball/%s' % (headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
         update_dir = os.path.join(headphones.PROG_DIR, 'update')
         version_path = os.path.join(headphones.PROG_DIR, 'version.txt')
 
@@ -192,7 +196,7 @@ def update():
             logger.error("Unable to retrieve new version from '%s', can't update", tar_download_url)
             return
 
-        download_name = headphones.GIT_BRANCH + '-github'
+        download_name = headphones.CONFIG.GIT_BRANCH + '-github'
         tar_download_path = os.path.join(headphones.PROG_DIR, download_name)
 
         # Save tar to disk
@@ -232,6 +236,8 @@ def update():
             with open(version_path, 'w') as f:
                 f.write(str(headphones.LATEST_VERSION))
         except IOError as e:
-            logger.error("Unable to write current version to version.txt, " \
+            logger.error(
-                "update not complete: ", e)
+                "Unable to write current version to version.txt, update not complete: %s",
+                e
+            )
             return

headphones/webstart.py

@@ -22,6 +22,7 @@ from headphones import logger
 from headphones.webserve import WebInterface
 from headphones.helpers import create_https_certificates
 
+
 def initialize(options=None):
     if options is None:
         options = {}
@@ -36,12 +37,12 @@ def initialize(options=None):
         if not (https_cert and os.path.exists(https_cert)) or not (https_key and os.path.exists(https_key)):
             if not create_https_certificates(https_cert, https_key):
                 logger.warn(u"Unable to create cert/key files, disabling HTTPS")
-                headphones.ENABLE_HTTPS = False
+                headphones.CONFIG.ENABLE_HTTPS = False
                 enable_https = False
 
         if not (os.path.exists(https_cert) and os.path.exists(https_key)):
             logger.warn(u"Disabled HTTPS because of missing CERT and KEY files")
-            headphones.ENABLE_HTTPS = False
+            headphones.CONFIG.ENABLE_HTTPS = False
             enable_https = False
 
     options_dict = {
@@ -52,7 +53,7 @@ def initialize(options=None):
         'tools.encode.encoding': 'utf-8',
         'tools.decode.on': True,
         'log.screen': False,
-        'engine.autoreload_on': False,
+        'engine.autoreload.on': False,
     }
 
     if enable_https:
@@ -94,7 +95,7 @@ def initialize(options=None):
         },
         '/cache': {
             'tools.staticdir.on': True,
-            'tools.staticdir.dir': headphones.CACHE_DIR
+            'tools.staticdir.dir': headphones.CONFIG.CACHE_DIR
         }
     }
 
@@ -104,20 +105,18 @@ def initialize(options=None):
         conf['/'].update({
             'tools.auth_basic.on': True,
             'tools.auth_basic.realm': 'Headphones web server',
-            'tools.auth_basic.checkpassword':  cherrypy.lib.auth_basic \
+            'tools.auth_basic.checkpassword': cherrypy.lib.auth_basic.checkpassword_dict({
-                .checkpassword_dict({
                 options['http_username']: options['http_password']
             })
         })
         conf['/api'] = {'tools.auth_basic.on': False}
 
-
     # Prevent time-outs
     cherrypy.engine.timeout_monitor.unsubscribe()
-    cherrypy.tree.mount(WebInterface(), options['http_root'], config=conf)
+    cherrypy.tree.mount(WebInterface(), str(options['http_root']), config=conf)
 
     try:
-        cherrypy.process.servers.check_port(options['http_host'], options['http_port'])
+        cherrypy.process.servers.check_port(str(options['http_host']), options['http_port'])
         cherrypy.server.start()
     except IOError:
         sys.stderr.write('Failed to start on port: %i. Is something else running?\n' % (options['http_port']))

lib/apscheduler/__init__.py

@@ -1,3 +1,5 @@
-version_info = (2, 0, 0, 'rc', 2)
+version_info = (3, 0, 1)
-version = '.'.join(str(n) for n in version_info[:3])
+version = '3.0.1'
-release = version + ''.join(str(n) for n in version_info[3:])
+release = '3.0.1'
+
+__version__ = release  # PEP 396

lib/apscheduler/events.py

@@ -1,63 +1,72 @@
-__all__ = ('EVENT_SCHEDULER_START', 'EVENT_SCHEDULER_SHUTDOWN',
+__all__ = ('EVENT_SCHEDULER_START', 'EVENT_SCHEDULER_SHUTDOWN', 'EVENT_EXECUTOR_ADDED', 'EVENT_EXECUTOR_REMOVED',
-           'EVENT_JOBSTORE_ADDED', 'EVENT_JOBSTORE_REMOVED',
+           'EVENT_JOBSTORE_ADDED', 'EVENT_JOBSTORE_REMOVED', 'EVENT_ALL_JOBS_REMOVED', 'EVENT_JOB_ADDED',
-           'EVENT_JOBSTORE_JOB_ADDED', 'EVENT_JOBSTORE_JOB_REMOVED',
+           'EVENT_JOB_REMOVED', 'EVENT_JOB_MODIFIED', 'EVENT_JOB_EXECUTED', 'EVENT_JOB_ERROR', 'EVENT_JOB_MISSED',
-           'EVENT_JOB_EXECUTED', 'EVENT_JOB_ERROR', 'EVENT_JOB_MISSED',
+           'SchedulerEvent', 'JobEvent', 'JobExecutionEvent')
-           'EVENT_ALL', 'SchedulerEvent', 'JobStoreEvent', 'JobEvent')
 
 
-EVENT_SCHEDULER_START = 1        # The scheduler was started
+EVENT_SCHEDULER_START = 1
-EVENT_SCHEDULER_SHUTDOWN = 2     # The scheduler was shut down
+EVENT_SCHEDULER_SHUTDOWN = 2
-EVENT_JOBSTORE_ADDED = 4         # A job store was added to the scheduler
+EVENT_EXECUTOR_ADDED = 4
-EVENT_JOBSTORE_REMOVED = 8       # A job store was removed from the scheduler
+EVENT_EXECUTOR_REMOVED = 8
-EVENT_JOBSTORE_JOB_ADDED = 16    # A job was added to a job store
+EVENT_JOBSTORE_ADDED = 16
-EVENT_JOBSTORE_JOB_REMOVED = 32  # A job was removed from a job store
+EVENT_JOBSTORE_REMOVED = 32
-EVENT_JOB_EXECUTED = 64          # A job was executed successfully
+EVENT_ALL_JOBS_REMOVED = 64
-EVENT_JOB_ERROR = 128            # A job raised an exception during execution
+EVENT_JOB_ADDED = 128
-EVENT_JOB_MISSED = 256           # A job's execution was missed
+EVENT_JOB_REMOVED = 256
-EVENT_ALL = (EVENT_SCHEDULER_START | EVENT_SCHEDULER_SHUTDOWN |
+EVENT_JOB_MODIFIED = 512
-             EVENT_JOBSTORE_ADDED | EVENT_JOBSTORE_REMOVED |
+EVENT_JOB_EXECUTED = 1024
-             EVENT_JOBSTORE_JOB_ADDED | EVENT_JOBSTORE_JOB_REMOVED |
+EVENT_JOB_ERROR = 2048
-             EVENT_JOB_EXECUTED | EVENT_JOB_ERROR | EVENT_JOB_MISSED)
+EVENT_JOB_MISSED = 4096
+EVENT_ALL = (EVENT_SCHEDULER_START | EVENT_SCHEDULER_SHUTDOWN | EVENT_JOBSTORE_ADDED | EVENT_JOBSTORE_REMOVED |
+             EVENT_JOB_ADDED | EVENT_JOB_REMOVED | EVENT_JOB_MODIFIED | EVENT_JOB_EXECUTED |
+             EVENT_JOB_ERROR | EVENT_JOB_MISSED)
 
 
 class SchedulerEvent(object):
     """
     An event that concerns the scheduler itself.
 
-    :var code: the type code of this event
+    :ivar code: the type code of this event
+    :ivar alias: alias of the job store or executor that was added or removed (if applicable)
     """
-    def __init__(self, code):
+
+    def __init__(self, code, alias=None):
+        super(SchedulerEvent, self).__init__()
         self.code = code
-
-
-class JobStoreEvent(SchedulerEvent):
-    """
-    An event that concerns job stores.
-
-    :var alias: the alias of the job store involved
-    :var job: the new job if a job was added
-    """
-    def __init__(self, code, alias, job=None):
-        SchedulerEvent.__init__(self, code)
         self.alias = alias
-        if job:
+
-            self.job = job
+    def __repr__(self):
+        return '<%s (code=%d)>' % (self.__class__.__name__, self.code)
 
 
 class JobEvent(SchedulerEvent):
+    """
+    An event that concerns a job.
+
+    :ivar code: the type code of this event
+    :ivar job_id: identifier of the job in question
+    :ivar jobstore: alias of the job store containing the job in question
+    """
+
+    def __init__(self, code, job_id, jobstore):
+        super(JobEvent, self).__init__(code)
+        self.code = code
+        self.job_id = job_id
+        self.jobstore = jobstore
+
+
+class JobExecutionEvent(JobEvent):
     """
     An event that concerns the execution of individual jobs.
 
-    :var job: the job instance in question
+    :ivar scheduled_run_time: the time when the job was scheduled to be run
-    :var scheduled_run_time: the time when the job was scheduled to be run
+    :ivar retval: the return value of the successfully executed job
-    :var retval: the return value of the successfully executed job
+    :ivar exception: the exception raised by the job
-    :var exception: the exception raised by the job
+    :ivar traceback: a formatted traceback for the exception
-    :var traceback: the traceback object associated with the exception
     """
-    def __init__(self, code, job, scheduled_run_time, retval=None,
+
-                 exception=None, traceback=None):
+    def __init__(self, code, job_id, jobstore, scheduled_run_time, retval=None, exception=None, traceback=None):
-        SchedulerEvent.__init__(self, code)
+        super(JobExecutionEvent, self).__init__(code, job_id, jobstore)
-        self.job = job
         self.scheduled_run_time = scheduled_run_time
         self.retval = retval
         self.exception = exception

lib/apscheduler/executors/asyncio.py

@@ -0,0 +1,28 @@
+from __future__ import absolute_import
+import sys
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class AsyncIOExecutor(BaseExecutor):
+    """
+    Runs jobs in the default executor of the event loop.
+
+    Plugin alias: ``asyncio``
+    """
+
+    def start(self, scheduler, alias):
+        super(AsyncIOExecutor, self).start(scheduler, alias)
+        self._eventloop = scheduler._eventloop
+
+    def _do_submit_job(self, job, run_times):
+        def callback(f):
+            try:
+                events = f.result()
+            except:
+                self._run_job_error(job.id, *sys.exc_info()[1:])
+            else:
+                self._run_job_success(job.id, events)
+
+        f = self._eventloop.run_in_executor(None, run_job, job, job._jobstore_alias, run_times, self._logger.name)
+        f.add_done_callback(callback)

lib/apscheduler/executors/base.py

@@ -0,0 +1,119 @@
+from abc import ABCMeta, abstractmethod
+from collections import defaultdict
+from datetime import datetime, timedelta
+from traceback import format_tb
+import logging
+import sys
+
+from pytz import utc
+import six
+
+from apscheduler.events import JobExecutionEvent, EVENT_JOB_MISSED, EVENT_JOB_ERROR, EVENT_JOB_EXECUTED
+
+
+class MaxInstancesReachedError(Exception):
+    def __init__(self, job):
+        super(MaxInstancesReachedError, self).__init__(
+            'Job "%s" has already reached its maximum number of instances (%d)' % (job.id, job.max_instances))
+
+
+class BaseExecutor(six.with_metaclass(ABCMeta, object)):
+    """Abstract base class that defines the interface that every executor must implement."""
+
+    _scheduler = None
+    _lock = None
+    _logger = logging.getLogger('apscheduler.executors')
+
+    def __init__(self):
+        super(BaseExecutor, self).__init__()
+        self._instances = defaultdict(lambda: 0)
+
+    def start(self, scheduler, alias):
+        """
+        Called by the scheduler when the scheduler is being started or when the executor is being added to an already
+        running scheduler.
+
+        :param apscheduler.schedulers.base.BaseScheduler scheduler: the scheduler that is starting this executor
+        :param str|unicode alias: alias of this executor as it was assigned to the scheduler
+        """
+
+        self._scheduler = scheduler
+        self._lock = scheduler._create_lock()
+        self._logger = logging.getLogger('apscheduler.executors.%s' % alias)
+
+    def shutdown(self, wait=True):
+        """
+        Shuts down this executor.
+
+        :param bool wait: ``True`` to wait until all submitted jobs have been executed
+        """
+
+    def submit_job(self, job, run_times):
+        """
+        Submits job for execution.
+
+        :param Job job: job to execute
+        :param list[datetime] run_times: list of datetimes specifying when the job should have been run
+        :raises MaxInstancesReachedError: if the maximum number of allowed instances for this job has been reached
+        """
+
+        assert self._lock is not None, 'This executor has not been started yet'
+        with self._lock:
+            if self._instances[job.id] >= job.max_instances:
+                raise MaxInstancesReachedError(job)
+
+            self._do_submit_job(job, run_times)
+            self._instances[job.id] += 1
+
+    @abstractmethod
+    def _do_submit_job(self, job, run_times):
+        """Performs the actual task of scheduling `run_job` to be called."""
+
+    def _run_job_success(self, job_id, events):
+        """Called by the executor with the list of generated events when `run_job` has been successfully called."""
+
+        with self._lock:
+            self._instances[job_id] -= 1
+
+        for event in events:
+            self._scheduler._dispatch_event(event)
+
+    def _run_job_error(self, job_id, exc, traceback=None):
+        """Called by the executor with the exception if there is an error calling `run_job`."""
+
+        with self._lock:
+            self._instances[job_id] -= 1
+
+        exc_info = (exc.__class__, exc, traceback)
+        self._logger.error('Error running job %s', job_id, exc_info=exc_info)
+
+
+def run_job(job, jobstore_alias, run_times, logger_name):
+    """Called by executors to run the job. Returns a list of scheduler events to be dispatched by the scheduler."""
+
+    events = []
+    logger = logging.getLogger(logger_name)
+    for run_time in run_times:
+        # See if the job missed its run time window, and handle possible misfires accordingly
+        if job.misfire_grace_time is not None:
+            difference = datetime.now(utc) - run_time
+            grace_time = timedelta(seconds=job.misfire_grace_time)
+            if difference > grace_time:
+                events.append(JobExecutionEvent(EVENT_JOB_MISSED, job.id, jobstore_alias, run_time))
+                logger.warning('Run time of job "%s" was missed by %s', job, difference)
+                continue
+
+        logger.info('Running job "%s" (scheduled at %s)', job, run_time)
+        try:
+            retval = job.func(*job.args, **job.kwargs)
+        except:
+            exc, tb = sys.exc_info()[1:]
+            formatted_tb = ''.join(format_tb(tb))
+            events.append(JobExecutionEvent(EVENT_JOB_ERROR, job.id, jobstore_alias, run_time, exception=exc,
+                                            traceback=formatted_tb))
+            logger.exception('Job "%s" raised an exception', job)
+        else:
+            events.append(JobExecutionEvent(EVENT_JOB_EXECUTED, job.id, jobstore_alias, run_time, retval=retval))
+            logger.info('Job "%s" executed successfully', job)
+
+    return events

lib/apscheduler/executors/debug.py

@@ -0,0 +1,19 @@
+import sys
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class DebugExecutor(BaseExecutor):
+    """
+    A special executor that executes the target callable directly instead of deferring it to a thread or process.
+
+    Plugin alias: ``debug``
+    """
+
+    def _do_submit_job(self, job, run_times):
+        try:
+            events = run_job(job, job._jobstore_alias, run_times, self._logger.name)
+        except:
+            self._run_job_error(job.id, *sys.exc_info()[1:])
+        else:
+            self._run_job_success(job.id, events)

lib/apscheduler/executors/gevent.py

@@ -0,0 +1,29 @@
+from __future__ import absolute_import
+import sys
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+try:
+    import gevent
+except ImportError:  # pragma: nocover
+    raise ImportError('GeventExecutor requires gevent installed')
+
+
+class GeventExecutor(BaseExecutor):
+    """
+    Runs jobs as greenlets.
+
+    Plugin alias: ``gevent``
+    """
+
+    def _do_submit_job(self, job, run_times):
+        def callback(greenlet):
+            try:
+                events = greenlet.get()
+            except:
+                self._run_job_error(job.id, *sys.exc_info()[1:])
+            else:
+                self._run_job_success(job.id, events)
+
+        gevent.spawn(run_job, job, job._jobstore_alias, run_times, self._logger.name).link(callback)

lib/apscheduler/executors/pool.py

@@ -0,0 +1,54 @@
+from abc import abstractmethod
+import concurrent.futures
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class BasePoolExecutor(BaseExecutor):
+    @abstractmethod
+    def __init__(self, pool):
+        super(BasePoolExecutor, self).__init__()
+        self._pool = pool
+
+    def _do_submit_job(self, job, run_times):
+        def callback(f):
+            exc, tb = (f.exception_info() if hasattr(f, 'exception_info') else
+                       (f.exception(), getattr(f.exception(), '__traceback__', None)))
+            if exc:
+                self._run_job_error(job.id, exc, tb)
+            else:
+                self._run_job_success(job.id, f.result())
+
+        f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
+        f.add_done_callback(callback)
+
+    def shutdown(self, wait=True):
+        self._pool.shutdown(wait)
+
+
+class ThreadPoolExecutor(BasePoolExecutor):
+    """
+    An executor that runs jobs in a concurrent.futures thread pool.
+
+    Plugin alias: ``threadpool``
+
+    :param max_workers: the maximum number of spawned threads.
+    """
+
+    def __init__(self, max_workers=10):
+        pool = concurrent.futures.ThreadPoolExecutor(int(max_workers))
+        super(ThreadPoolExecutor, self).__init__(pool)
+
+
+class ProcessPoolExecutor(BasePoolExecutor):
+    """
+    An executor that runs jobs in a concurrent.futures process pool.
+
+    Plugin alias: ``processpool``
+
+    :param max_workers: the maximum number of spawned processes.
+    """
+
+    def __init__(self, max_workers=10):
+        pool = concurrent.futures.ProcessPoolExecutor(int(max_workers))
+        super(ProcessPoolExecutor, self).__init__(pool)

lib/apscheduler/executors/twisted.py

@@ -0,0 +1,25 @@
+from __future__ import absolute_import
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class TwistedExecutor(BaseExecutor):
+    """
+    Runs jobs in the reactor's thread pool.
+
+    Plugin alias: ``twisted``
+    """
+
+    def start(self, scheduler, alias):
+        super(TwistedExecutor, self).start(scheduler, alias)
+        self._reactor = scheduler._reactor
+
+    def _do_submit_job(self, job, run_times):
+        def callback(success, result):
+            if success:
+                self._run_job_success(job.id, result)
+            else:
+                self._run_job_error(job.id, result.value, result.tb)
+
+        self._reactor.getThreadPool().callInThreadWithCallback(callback, run_job, job, job._jobstore_alias, run_times,
+                                                               self._logger.name)

lib/apscheduler/job.py

@@ -1,134 +1,252 @@
-"""
+from collections import Iterable, Mapping
-Jobs represent scheduled tasks.
+from uuid import uuid4
-"""
 
-from threading import Lock
+import six
-from datetime import timedelta
 
-from apscheduler.util import to_unicode, ref_to_obj, get_callable_name,\
+from apscheduler.triggers.base import BaseTrigger
-    obj_to_ref
+from apscheduler.util import ref_to_obj, obj_to_ref, datetime_repr, repr_escape, get_callable_name, check_callable_args, \
-
+    convert_to_datetime
-
-class MaxInstancesReachedError(Exception):
-    pass
 
 
 class Job(object):
     """
-    Encapsulates the actual Job along with its metadata. Job instances
+    Contains the options given when scheduling callables and its current schedule and other state.
-    are created by the scheduler when adding jobs, and it should not be
+    This class should never be instantiated by the user.
-    directly instantiated.
 
-    :param trigger: trigger that determines the execution times
+    :var str id: the unique identifier of this job
-    :param func: callable to call when the trigger is triggered
+    :var str name: the description of this job
-    :param args: list of positional arguments to call func with
+    :var func: the callable to execute
-    :param kwargs: dict of keyword arguments to call func with
+    :var tuple|list args: positional arguments to the callable
-    :param name: name of the job (optional)
+    :var dict kwargs: keyword arguments to the callable
-    :param misfire_grace_time: seconds after the designated run time that
+    :var bool coalesce: whether to only run the job once when several run times are due
-        the job is still allowed to be run
+    :var trigger: the trigger object that controls the schedule of this job
-    :param coalesce: run once instead of many times if the scheduler determines
+    :var str executor: the name of the executor that will run this job
-        that the job should be run more than once in succession
+    :var int misfire_grace_time: the time (in seconds) how much this job's execution is allowed to be late
-    :param max_runs: maximum number of times this job is allowed to be
+    :var int max_instances: the maximum number of concurrently executing instances allowed for this job
-        triggered
+    :var datetime.datetime next_run_time: the next scheduled run time of this job
-    :param max_instances: maximum number of concurrently running
-        instances allowed for this job
     """
-    id = None
-    next_run_time = None
 
-    def __init__(self, trigger, func, args, kwargs, misfire_grace_time,
+    __slots__ = ('_scheduler', '_jobstore_alias', 'id', 'trigger', 'executor', 'func', 'func_ref', 'args', 'kwargs',
-                 coalesce, name=None, max_runs=None, max_instances=1):
+                 'name', 'misfire_grace_time', 'coalesce', 'max_instances', 'next_run_time')
-        if not trigger:
-            raise ValueError('The trigger must not be None')
-        if not hasattr(func, '__call__'):
-            raise TypeError('func must be callable')
-        if not hasattr(args, '__getitem__'):
-            raise TypeError('args must be a list-like object')
-        if not hasattr(kwargs, '__getitem__'):
-            raise TypeError('kwargs must be a dict-like object')
-        if misfire_grace_time <= 0:
-            raise ValueError('misfire_grace_time must be a positive value')
-        if max_runs is not None and max_runs <= 0:
-            raise ValueError('max_runs must be a positive value')
-        if max_instances <= 0:
-            raise ValueError('max_instances must be a positive value')
 
-        self._lock = Lock()
+    def __init__(self, scheduler, id=None, **kwargs):
+        super(Job, self).__init__()
+        self._scheduler = scheduler
+        self._jobstore_alias = None
+        self._modify(id=id or uuid4().hex, **kwargs)
 
-        self.trigger = trigger
+    def modify(self, **changes):
-        self.func = func
-        self.args = args
-        self.kwargs = kwargs
-        self.name = to_unicode(name or get_callable_name(func))
-        self.misfire_grace_time = misfire_grace_time
-        self.coalesce = coalesce
-        self.max_runs = max_runs
-        self.max_instances = max_instances
-        self.runs = 0
-        self.instances = 0
-
-    def compute_next_run_time(self, now):
-        if self.runs == self.max_runs:
-            self.next_run_time = None
-        else:
-            self.next_run_time = self.trigger.get_next_fire_time(now)
-
-        return self.next_run_time
-
-    def get_run_times(self, now):
         """
-        Computes the scheduled run times between ``next_run_time`` and ``now``.
+        Makes the given changes to this job and saves it in the associated job store.
+        Accepted keyword arguments are the same as the variables on this class.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.modify_job`
         """
+
+        self._scheduler.modify_job(self.id, self._jobstore_alias, **changes)
+
+    def reschedule(self, trigger, **trigger_args):
+        """
+        Shortcut for switching the trigger on this job.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.reschedule_job`
+        """
+
+        self._scheduler.reschedule_job(self.id, self._jobstore_alias, trigger, **trigger_args)
+
+    def pause(self):
+        """
+        Temporarily suspend the execution of this job.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.pause_job`
+        """
+
+        self._scheduler.pause_job(self.id, self._jobstore_alias)
+
+    def resume(self):
+        """
+        Resume the schedule of this job if previously paused.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.resume_job`
+        """
+
+        self._scheduler.resume_job(self.id, self._jobstore_alias)
+
+    def remove(self):
+        """
+        Unschedules this job and removes it from its associated job store.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.remove_job`
+        """
+
+        self._scheduler.remove_job(self.id, self._jobstore_alias)
+
+    @property
+    def pending(self):
+        """Returns ``True`` if the referenced job is still waiting to be added to its designated job store."""
+
+        return self._jobstore_alias is None
+
+    #
+    # Private API
+    #
+
+    def _get_run_times(self, now):
+        """
+        Computes the scheduled run times between ``next_run_time`` and ``now`` (inclusive).
+
+        :type now: datetime.datetime
+        :rtype: list[datetime.datetime]
+        """
+
         run_times = []
-        run_time = self.next_run_time
+        next_run_time = self.next_run_time
-        increment = timedelta(microseconds=1)
+        while next_run_time and next_run_time <= now:
-        while ((not self.max_runs or self.runs < self.max_runs) and
+            run_times.append(next_run_time)
-               run_time and run_time <= now):
+            next_run_time = self.trigger.get_next_fire_time(next_run_time, now)
-            run_times.append(run_time)
-            run_time = self.trigger.get_next_fire_time(run_time + increment)
 
         return run_times
 
-    def add_instance(self):
+    def _modify(self, **changes):
-        self._lock.acquire()
+        """Validates the changes to the Job and makes the modifications if and only if all of them validate."""
-        try:
-            if self.instances == self.max_instances:
-                raise MaxInstancesReachedError
-            self.instances += 1
-        finally:
-            self._lock.release()
 
-    def remove_instance(self):
+        approved = {}
-        self._lock.acquire()
+
+        if 'id' in changes:
+            value = changes.pop('id')
+            if not isinstance(value, six.string_types):
+                raise TypeError("id must be a nonempty string")
+            if hasattr(self, 'id'):
+                raise ValueError('The job ID may not be changed')
+            approved['id'] = value
+
+        if 'func' in changes or 'args' in changes or 'kwargs' in changes:
+            func = changes.pop('func') if 'func' in changes else self.func
+            args = changes.pop('args') if 'args' in changes else self.args
+            kwargs = changes.pop('kwargs') if 'kwargs' in changes else self.kwargs
+
+            if isinstance(func, str):
+                func_ref = func
+                func = ref_to_obj(func)
+            elif callable(func):
                 try:
-            assert self.instances > 0, 'Already at 0 instances'
+                    func_ref = obj_to_ref(func)
-            self.instances -= 1
+                except ValueError:
-        finally:
+                    # If this happens, this Job won't be serializable
-            self._lock.release()
+                    func_ref = None
+            else:
+                raise TypeError('func must be a callable or a textual reference to one')
+
+            if not hasattr(self, 'name') and changes.get('name', None) is None:
+                changes['name'] = get_callable_name(func)
+
+            if isinstance(args, six.string_types) or not isinstance(args, Iterable):
+                raise TypeError('args must be a non-string iterable')
+            if isinstance(kwargs, six.string_types) or not isinstance(kwargs, Mapping):
+                raise TypeError('kwargs must be a dict-like object')
+
+            check_callable_args(func, args, kwargs)
+
+            approved['func'] = func
+            approved['func_ref'] = func_ref
+            approved['args'] = args
+            approved['kwargs'] = kwargs
+
+        if 'name' in changes:
+            value = changes.pop('name')
+            if not value or not isinstance(value, six.string_types):
+                raise TypeError("name must be a nonempty string")
+            approved['name'] = value
+
+        if 'misfire_grace_time' in changes:
+            value = changes.pop('misfire_grace_time')
+            if value is not None and (not isinstance(value, six.integer_types) or value <= 0):
+                raise TypeError('misfire_grace_time must be either None or a positive integer')
+            approved['misfire_grace_time'] = value
+
+        if 'coalesce' in changes:
+            value = bool(changes.pop('coalesce'))
+            approved['coalesce'] = value
+
+        if 'max_instances' in changes:
+            value = changes.pop('max_instances')
+            if not isinstance(value, six.integer_types) or value <= 0:
+                raise TypeError('max_instances must be a positive integer')
+            approved['max_instances'] = value
+
+        if 'trigger' in changes:
+            trigger = changes.pop('trigger')
+            if not isinstance(trigger, BaseTrigger):
+                raise TypeError('Expected a trigger instance, got %s instead' % trigger.__class__.__name__)
+
+            approved['trigger'] = trigger
+
+        if 'executor' in changes:
+            value = changes.pop('executor')
+            if not isinstance(value, six.string_types):
+                raise TypeError('executor must be a string')
+            approved['executor'] = value
+
+        if 'next_run_time' in changes:
+            value = changes.pop('next_run_time')
+            approved['next_run_time'] = convert_to_datetime(value, self._scheduler.timezone, 'next_run_time')
+
+        if changes:
+            raise AttributeError('The following are not modifiable attributes of Job: %s' % ', '.join(changes))
+
+        for key, value in six.iteritems(approved):
+            setattr(self, key, value)
 
     def __getstate__(self):
-        # Prevents the unwanted pickling of transient or unpicklable variables
+        # Don't allow this Job to be serialized if the function reference could not be determined
-        state = self.__dict__.copy()
+        if not self.func_ref:
-        state.pop('instances', None)
+            raise ValueError('This Job cannot be serialized since the reference to its callable (%r) could not be '
-        state.pop('func', None)
+                             'determined. Consider giving a textual reference (module:function name) instead.' %
-        state.pop('_lock', None)
+                             (self.func,))
-        state['func_ref'] = obj_to_ref(self.func)
+
-        return state
+        return {
+            'version': 1,
+            'id': self.id,
+            'func': self.func_ref,
+            'trigger': self.trigger,
+            'executor': self.executor,
+            'args': self.args,
+            'kwargs': self.kwargs,
+            'name': self.name,
+            'misfire_grace_time': self.misfire_grace_time,
+            'coalesce': self.coalesce,
+            'max_instances': self.max_instances,
+            'next_run_time': self.next_run_time
+        }
 
     def __setstate__(self, state):
-        state['instances'] = 0
+        if state.get('version', 1) > 1:
-        state['func'] = ref_to_obj(state.pop('func_ref'))
+            raise ValueError('Job has version %s, but only version 1 can be handled' % state['version'])
-        state['_lock'] = Lock()
+
-        self.__dict__ = state
+        self.id = state['id']
+        self.func_ref = state['func']
+        self.func = ref_to_obj(self.func_ref)
+        self.trigger = state['trigger']
+        self.executor = state['executor']
+        self.args = state['args']
+        self.kwargs = state['kwargs']
+        self.name = state['name']
+        self.misfire_grace_time = state['misfire_grace_time']
+        self.coalesce = state['coalesce']
+        self.max_instances = state['max_instances']
+        self.next_run_time = state['next_run_time']
 
     def __eq__(self, other):
         if isinstance(other, Job):
-            return self.id is not None and other.id == self.id or self is other
+            return self.id == other.id
         return NotImplemented
 
     def __repr__(self):
-        return '<Job (name=%s, trigger=%s)>' % (self.name, repr(self.trigger))
+        return '<Job (id=%s name=%s)>' % (repr_escape(self.id), repr_escape(self.name))
 
     def __str__(self):
-        return '%s (trigger: %s, next run at: %s)' % (self.name,
+        return '%s (trigger: %s, next run at: %s)' % (repr_escape(self.name), repr_escape(str(self.trigger)),
-            str(self.trigger), str(self.next_run_time))
+                                                      datetime_repr(self.next_run_time))
+
+    def __unicode__(self):
+        return six.u('%s (trigger: %s, next run at: %s)') % (self.name, self.trigger, datetime_repr(self.next_run_time))

lib/apscheduler/jobstores/base.py

@@ -1,25 +1,127 @@
+from abc import ABCMeta, abstractmethod
+import logging
+
+import six
+
+
+class JobLookupError(KeyError):
+    """Raised when the job store cannot find a job for update or removal."""
+
+    def __init__(self, job_id):
+        super(JobLookupError, self).__init__(six.u('No job by the id of %s was found') % job_id)
+
+
+class ConflictingIdError(KeyError):
+    """Raised when the uniqueness of job IDs is being violated."""
+
+    def __init__(self, job_id):
+        super(ConflictingIdError, self).__init__(six.u('Job identifier (%s) conflicts with an existing job') % job_id)
+
+
+class TransientJobError(ValueError):
+    """Raised when an attempt to add transient (with no func_ref) job to a persistent job store is detected."""
+
+    def __init__(self, job_id):
+        super(TransientJobError, self).__init__(
+            six.u('Job (%s) cannot be added to this job store because a reference to the callable could not be '
+                  'determined.') % job_id)
+
+
+class BaseJobStore(six.with_metaclass(ABCMeta)):
+    """Abstract base class that defines the interface that every job store must implement."""
+
+    _scheduler = None
+    _alias = None
+    _logger = logging.getLogger('apscheduler.jobstores')
+
+    def start(self, scheduler, alias):
         """
-Abstract base class that provides the interface needed by all job stores.
+        Called by the scheduler when the scheduler is being started or when the job store is being added to an already
-Job store methods are also documented here.
+        running scheduler.
+
+        :param apscheduler.schedulers.base.BaseScheduler scheduler: the scheduler that is starting this job store
+        :param str|unicode alias: alias of this job store as it was assigned to the scheduler
         """
 
+        self._scheduler = scheduler
+        self._alias = alias
+        self._logger = logging.getLogger('apscheduler.jobstores.%s' % alias)
 
-class JobStore(object):
+    def shutdown(self):
-    def add_job(self, job):
-        """Adds the given job from this store."""
-        raise NotImplementedError
-
-    def update_job(self, job):
-        """Persists the running state of the given job."""
-        raise NotImplementedError
-
-    def remove_job(self, job):
-        """Removes the given jobs from this store."""
-        raise NotImplementedError
-
-    def load_jobs(self):
-        """Loads jobs from this store into memory."""
-        raise NotImplementedError
-
-    def close(self):
         """Frees any resources still bound to this job store."""
+
+    @abstractmethod
+    def lookup_job(self, job_id):
+        """
+        Returns a specific job, or ``None`` if it isn't found..
+
+        The job store is responsible for setting the ``scheduler`` and ``jobstore`` attributes of the returned job to
+        point to the scheduler and itself, respectively.
+
+        :param str|unicode job_id: identifier of the job
+        :rtype: Job
+        """
+
+    @abstractmethod
+    def get_due_jobs(self, now):
+        """
+        Returns the list of jobs that have ``next_run_time`` earlier or equal to ``now``.
+        The returned jobs must be sorted by next run time (ascending).
+
+        :param datetime.datetime now: the current (timezone aware) datetime
+        :rtype: list[Job]
+        """
+
+    @abstractmethod
+    def get_next_run_time(self):
+        """
+        Returns the earliest run time of all the jobs stored in this job store, or ``None`` if there are no active jobs.
+
+        :rtype: datetime.datetime
+        """
+
+    @abstractmethod
+    def get_all_jobs(self):
+        """
+        Returns a list of all jobs in this job store. The returned jobs should be sorted by next run time (ascending).
+        Paused jobs (next_run_time is None) should be sorted last.
+
+        The job store is responsible for setting the ``scheduler`` and ``jobstore`` attributes of the returned jobs to
+        point to the scheduler and itself, respectively.
+
+        :rtype: list[Job]
+        """
+
+    @abstractmethod
+    def add_job(self, job):
+        """
+        Adds the given job to this store.
+
+        :param Job job: the job to add
+        :raises ConflictingIdError: if there is another job in this store with the same ID
+        """
+
+    @abstractmethod
+    def update_job(self, job):
+        """
+        Replaces the job in the store with the given newer version.
+
+        :param Job job: the job to update
+        :raises JobLookupError: if the job does not exist
+        """
+
+    @abstractmethod
+    def remove_job(self, job_id):
+        """
+        Removes the given job from this store.
+
+        :param str|unicode job_id: identifier of the job
+        :raises JobLookupError: if the job does not exist
+        """
+
+    @abstractmethod
+    def remove_all_jobs(self):
+        """Removes all jobs from this store."""
+
+    def __repr__(self):
+        return '<%s>' % self.__class__.__name__

lib/apscheduler/jobstores/memory.py

@@ -0,0 +1,107 @@
+from __future__ import absolute_import
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import datetime_to_utc_timestamp
+
+
+class MemoryJobStore(BaseJobStore):
+    """
+    Stores jobs in an array in RAM. Provides no persistence support.
+
+    Plugin alias: ``memory``
+    """
+
+    def __init__(self):
+        super(MemoryJobStore, self).__init__()
+        self._jobs = []  # list of (job, timestamp), sorted by next_run_time and job id (ascending)
+        self._jobs_index = {}  # id -> (job, timestamp) lookup table
+
+    def lookup_job(self, job_id):
+        return self._jobs_index.get(job_id, (None, None))[0]
+
+    def get_due_jobs(self, now):
+        now_timestamp = datetime_to_utc_timestamp(now)
+        pending = []
+        for job, timestamp in self._jobs:
+            if timestamp is None or timestamp > now_timestamp:
+                break
+            pending.append(job)
+
+        return pending
+
+    def get_next_run_time(self):
+        return self._jobs[0][0].next_run_time if self._jobs else None
+
+    def get_all_jobs(self):
+        return [j[0] for j in self._jobs]
+
+    def add_job(self, job):
+        if job.id in self._jobs_index:
+            raise ConflictingIdError(job.id)
+
+        timestamp = datetime_to_utc_timestamp(job.next_run_time)
+        index = self._get_job_index(timestamp, job.id)
+        self._jobs.insert(index, (job, timestamp))
+        self._jobs_index[job.id] = (job, timestamp)
+
+    def update_job(self, job):
+        old_job, old_timestamp = self._jobs_index.get(job.id, (None, None))
+        if old_job is None:
+            raise JobLookupError(job.id)
+
+        # If the next run time has not changed, simply replace the job in its present index.
+        # Otherwise, reinsert the job to the list to preserve the ordering.
+        old_index = self._get_job_index(old_timestamp, old_job.id)
+        new_timestamp = datetime_to_utc_timestamp(job.next_run_time)
+        if old_timestamp == new_timestamp:
+            self._jobs[old_index] = (job, new_timestamp)
+        else:
+            del self._jobs[old_index]
+            new_index = self._get_job_index(new_timestamp, job.id)
+            self._jobs.insert(new_index, (job, new_timestamp))
+
+        self._jobs_index[old_job.id] = (job, new_timestamp)
+
+    def remove_job(self, job_id):
+        job, timestamp = self._jobs_index.get(job_id, (None, None))
+        if job is None:
+            raise JobLookupError(job_id)
+
+        index = self._get_job_index(timestamp, job_id)
+        del self._jobs[index]
+        del self._jobs_index[job.id]
+
+    def remove_all_jobs(self):
+        self._jobs = []
+        self._jobs_index = {}
+
+    def shutdown(self):
+        self.remove_all_jobs()
+
+    def _get_job_index(self, timestamp, job_id):
+        """
+        Returns the index of the given job, or if it's not found, the index where the job should be inserted based on
+        the given timestamp.
+
+        :type timestamp: int
+        :type job_id: str
+        """
+
+        lo, hi = 0, len(self._jobs)
+        timestamp = float('inf') if timestamp is None else timestamp
+        while lo < hi:
+            mid = (lo + hi) // 2
+            mid_job, mid_timestamp = self._jobs[mid]
+            mid_timestamp = float('inf') if mid_timestamp is None else mid_timestamp
+            if mid_timestamp > timestamp:
+                hi = mid
+            elif mid_timestamp < timestamp:
+                lo = mid + 1
+            elif mid_job.id > job_id:
+                hi = mid
+            elif mid_job.id < job_id:
+                lo = mid + 1
+            else:
+                return mid
+
+        return lo

lib/apscheduler/jobstores/mongodb.py

@@ -0,0 +1,124 @@
+from __future__ import absolute_import
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import maybe_ref, datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from bson.binary import Binary
+    from pymongo.errors import DuplicateKeyError
+    from pymongo import MongoClient, ASCENDING
+except ImportError:  # pragma: nocover
+    raise ImportError('MongoDBJobStore requires PyMongo installed')
+
+
+class MongoDBJobStore(BaseJobStore):
+    """
+    Stores jobs in a MongoDB database. Any leftover keyword arguments are directly passed to pymongo's `MongoClient
+    <http://api.mongodb.org/python/current/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient>`_.
+
+    Plugin alias: ``mongodb``
+
+    :param str database: database to store jobs in
+    :param str collection: collection to store jobs in
+    :param client: a :class:`~pymongo.mongo_client.MongoClient` instance to use instead of providing connection
+                   arguments
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    """
+
+    def __init__(self, database='apscheduler', collection='jobs', client=None,
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL, **connect_args):
+        super(MongoDBJobStore, self).__init__()
+        self.pickle_protocol = pickle_protocol
+
+        if not database:
+            raise ValueError('The "database" parameter must not be empty')
+        if not collection:
+            raise ValueError('The "collection" parameter must not be empty')
+
+        if client:
+            self.connection = maybe_ref(client)
+        else:
+            connect_args.setdefault('w', 1)
+            self.connection = MongoClient(**connect_args)
+
+        self.collection = self.connection[database][collection]
+        self.collection.ensure_index('next_run_time', sparse=True)
+
+    def lookup_job(self, job_id):
+        document = self.collection.find_one(job_id, ['job_state'])
+        return self._reconstitute_job(document['job_state']) if document else None
+
+    def get_due_jobs(self, now):
+        timestamp = datetime_to_utc_timestamp(now)
+        return self._get_jobs({'next_run_time': {'$lte': timestamp}})
+
+    def get_next_run_time(self):
+        document = self.collection.find_one({'next_run_time': {'$ne': None}}, fields=['next_run_time'],
+                                            sort=[('next_run_time', ASCENDING)])
+        return utc_timestamp_to_datetime(document['next_run_time']) if document else None
+
+    def get_all_jobs(self):
+        return self._get_jobs({})
+
+    def add_job(self, job):
+        try:
+            self.collection.insert({
+                '_id': job.id,
+                'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+                'job_state': Binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
+            })
+        except DuplicateKeyError:
+            raise ConflictingIdError(job.id)
+
+    def update_job(self, job):
+        changes = {
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': Binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
+        }
+        result = self.collection.update({'_id': job.id}, {'$set': changes})
+        if result and result['n'] == 0:
+            raise JobLookupError(id)
+
+    def remove_job(self, job_id):
+        result = self.collection.remove(job_id)
+        if result and result['n'] == 0:
+            raise JobLookupError(job_id)
+
+    def remove_all_jobs(self):
+        self.collection.remove()
+
+    def shutdown(self):
+        self.connection.disconnect()
+
+    def _reconstitute_job(self, job_state):
+        job_state = pickle.loads(job_state)
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _get_jobs(self, conditions):
+        jobs = []
+        failed_job_ids = []
+        for document in self.collection.find(conditions, ['_id', 'job_state'], sort=[('next_run_time', ASCENDING)]):
+            try:
+                jobs.append(self._reconstitute_job(document['job_state']))
+            except:
+                self._logger.exception('Unable to restore job "%s" -- removing it', document['_id'])
+                failed_job_ids.append(document['_id'])
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            self.collection.remove({'_id': {'$in': failed_job_ids}})
+
+        return jobs
+
+    def __repr__(self):
+        return '<%s (client=%s)>' % (self.__class__.__name__, self.connection)

lib/apscheduler/jobstores/mongodb_store.py

@@ -1,84 +0,0 @@
-"""
-Stores jobs in a MongoDB database.
-"""
-import logging
-
-from apscheduler.jobstores.base import JobStore
-from apscheduler.job import Job
-
-try:
-    import cPickle as pickle
-except ImportError:  # pragma: nocover
-    import pickle
-
-try:
-    from bson.binary import Binary
-    from pymongo.connection import Connection
-except ImportError:  # pragma: nocover
-    raise ImportError('MongoDBJobStore requires PyMongo installed')
-
-logger = logging.getLogger(__name__)
-
-
-class MongoDBJobStore(JobStore):
-    def __init__(self, database='apscheduler', collection='jobs',
-                 connection=None, pickle_protocol=pickle.HIGHEST_PROTOCOL,
-                 **connect_args):
-        self.jobs = []
-        self.pickle_protocol = pickle_protocol
-
-        if not database:
-            raise ValueError('The "database" parameter must not be empty')
-        if not collection:
-            raise ValueError('The "collection" parameter must not be empty')
-
-        if connection:
-            self.connection = connection
-        else:
-            self.connection = Connection(**connect_args)
-
-        self.collection = self.connection[database][collection]
-
-    def add_job(self, job):
-        job_dict = job.__getstate__()
-        job_dict['trigger'] = Binary(pickle.dumps(job.trigger,
-                                                  self.pickle_protocol))
-        job_dict['args'] = Binary(pickle.dumps(job.args,
-                                               self.pickle_protocol))
-        job_dict['kwargs'] = Binary(pickle.dumps(job.kwargs,
-                                                 self.pickle_protocol))
-        job.id = self.collection.insert(job_dict)
-        self.jobs.append(job)
-
-    def remove_job(self, job):
-        self.collection.remove(job.id)
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        jobs = []
-        for job_dict in self.collection.find():
-            try:
-                job = Job.__new__(Job)
-                job_dict['id'] = job_dict.pop('_id')
-                job_dict['trigger'] = pickle.loads(job_dict['trigger'])
-                job_dict['args'] = pickle.loads(job_dict['args'])
-                job_dict['kwargs'] = pickle.loads(job_dict['kwargs'])
-                job.__setstate__(job_dict)
-                jobs.append(job)
-            except Exception:
-                job_name = job_dict.get('name', '(unknown)')
-                logger.exception('Unable to restore job "%s"', job_name)
-        self.jobs = jobs
-
-    def update_job(self, job):
-        spec = {'_id': job.id}
-        document = {'$set': {'next_run_time': job.next_run_time},
-                    '$inc': {'runs': 1}}
-        self.collection.update(spec, document)
-
-    def close(self):
-        self.connection.disconnect()
-
-    def __repr__(self):
-        connection = self.collection.database.connection
-        return '<%s (connection=%s)>' % (self.__class__.__name__, connection)

lib/apscheduler/jobstores/ram_store.py

@@ -1,25 +0,0 @@
-"""
-Stores jobs in an array in RAM. Provides no persistence support.
-"""
-
-from apscheduler.jobstores.base import JobStore
-
-
-class RAMJobStore(JobStore):
-    def __init__(self):
-        self.jobs = []
-
-    def add_job(self, job):
-        self.jobs.append(job)
-
-    def update_job(self, job):
-        pass
-
-    def remove_job(self, job):
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        pass
-
-    def __repr__(self):
-        return '<%s>' % (self.__class__.__name__)

lib/apscheduler/jobstores/redis.py

@@ -0,0 +1,138 @@
+from __future__ import absolute_import
+
+import six
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from redis import StrictRedis
+except ImportError:  # pragma: nocover
+    raise ImportError('RedisJobStore requires redis installed')
+
+
+class RedisJobStore(BaseJobStore):
+    """
+    Stores jobs in a Redis database. Any leftover keyword arguments are directly passed to redis's StrictRedis.
+
+    Plugin alias: ``redis``
+
+    :param int db: the database number to store jobs in
+    :param str jobs_key: key to store jobs in
+    :param str run_times_key: key to store the jobs' run times in
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    """
+
+    def __init__(self, db=0, jobs_key='apscheduler.jobs', run_times_key='apscheduler.run_times',
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL, **connect_args):
+        super(RedisJobStore, self).__init__()
+
+        if db is None:
+            raise ValueError('The "db" parameter must not be empty')
+        if not jobs_key:
+            raise ValueError('The "jobs_key" parameter must not be empty')
+        if not run_times_key:
+            raise ValueError('The "run_times_key" parameter must not be empty')
+
+        self.pickle_protocol = pickle_protocol
+        self.jobs_key = jobs_key
+        self.run_times_key = run_times_key
+        self.redis = StrictRedis(db=int(db), **connect_args)
+
+    def lookup_job(self, job_id):
+        job_state = self.redis.hget(self.jobs_key, job_id)
+        return self._reconstitute_job(job_state) if job_state else None
+
+    def get_due_jobs(self, now):
+        timestamp = datetime_to_utc_timestamp(now)
+        job_ids = self.redis.zrangebyscore(self.run_times_key, 0, timestamp)
+        if job_ids:
+            job_states = self.redis.hmget(self.jobs_key, *job_ids)
+            return self._reconstitute_jobs(six.moves.zip(job_ids, job_states))
+        return []
+
+    def get_next_run_time(self):
+        next_run_time = self.redis.zrange(self.run_times_key, 0, 0, withscores=True)
+        if next_run_time:
+            return utc_timestamp_to_datetime(next_run_time[0][1])
+
+    def get_all_jobs(self):
+        job_states = self.redis.hgetall(self.jobs_key)
+        jobs = self._reconstitute_jobs(six.iteritems(job_states))
+        return sorted(jobs, key=lambda job: job.next_run_time)
+
+    def add_job(self, job):
+        if self.redis.hexists(self.jobs_key, job.id):
+            raise ConflictingIdError(job.id)
+
+        with self.redis.pipeline() as pipe:
+            pipe.multi()
+            pipe.hset(self.jobs_key, job.id, pickle.dumps(job.__getstate__(), self.pickle_protocol))
+            pipe.zadd(self.run_times_key, datetime_to_utc_timestamp(job.next_run_time), job.id)
+            pipe.execute()
+
+    def update_job(self, job):
+        if not self.redis.hexists(self.jobs_key, job.id):
+            raise JobLookupError(job.id)
+
+        with self.redis.pipeline() as pipe:
+            pipe.hset(self.jobs_key, job.id, pickle.dumps(job.__getstate__(), self.pickle_protocol))
+            if job.next_run_time:
+                pipe.zadd(self.run_times_key, datetime_to_utc_timestamp(job.next_run_time), job.id)
+            else:
+                pipe.zrem(self.run_times_key, job.id)
+            pipe.execute()
+
+    def remove_job(self, job_id):
+        if not self.redis.hexists(self.jobs_key, job_id):
+            raise JobLookupError(job_id)
+
+        with self.redis.pipeline() as pipe:
+            pipe.hdel(self.jobs_key, job_id)
+            pipe.zrem(self.run_times_key, job_id)
+            pipe.execute()
+
+    def remove_all_jobs(self):
+        with self.redis.pipeline() as pipe:
+            pipe.delete(self.jobs_key)
+            pipe.delete(self.run_times_key)
+            pipe.execute()
+
+    def shutdown(self):
+        self.redis.connection_pool.disconnect()
+
+    def _reconstitute_job(self, job_state):
+        job_state = pickle.loads(job_state)
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _reconstitute_jobs(self, job_states):
+        jobs = []
+        failed_job_ids = []
+        for job_id, job_state in job_states:
+            try:
+                jobs.append(self._reconstitute_job(job_state))
+            except:
+                self._logger.exception('Unable to restore job "%s" -- removing it', job_id)
+                failed_job_ids.append(job_id)
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            with self.redis.pipeline() as pipe:
+                pipe.hdel(self.jobs_key, *failed_job_ids)
+                pipe.zrem(self.run_times_key, *failed_job_ids)
+                pipe.execute()
+
+        return jobs
+
+    def __repr__(self):
+        return '<%s>' % self.__class__.__name__

lib/apscheduler/jobstores/shelve_store.py

@@ -1,65 +0,0 @@
-"""
-Stores jobs in a file governed by the :mod:`shelve` module.
-"""
-
-import shelve
-import pickle
-import random
-import logging
-
-from apscheduler.jobstores.base import JobStore
-from apscheduler.job import Job
-from apscheduler.util import itervalues
-
-logger = logging.getLogger(__name__)
-
-
-class ShelveJobStore(JobStore):
-    MAX_ID = 1000000
-
-    def __init__(self, path, pickle_protocol=pickle.HIGHEST_PROTOCOL):
-        self.jobs = []
-        self.path = path
-        self.pickle_protocol = pickle_protocol
-        self.store = shelve.open(path, 'c', self.pickle_protocol)
-
-    def _generate_id(self):
-        id = None
-        while not id:
-            id = str(random.randint(1, self.MAX_ID))
-            if not id in self.store:
-                return id
-
-    def add_job(self, job):
-        job.id = self._generate_id()
-        self.jobs.append(job)
-        self.store[job.id] = job.__getstate__()
-
-    def update_job(self, job):
-        job_dict = self.store[job.id]
-        job_dict['next_run_time'] = job.next_run_time
-        job_dict['runs'] = job.runs
-        self.store[job.id] = job_dict
-
-    def remove_job(self, job):
-        del self.store[job.id]
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        jobs = []
-        for job_dict in itervalues(self.store):
-            try:
-                job = Job.__new__(Job)
-                job.__setstate__(job_dict)
-                jobs.append(job)
-            except Exception:
-                job_name = job_dict.get('name', '(unknown)')
-                logger.exception('Unable to restore job "%s"', job_name)
-
-        self.jobs = jobs
-
-    def close(self):
-        self.store.close()
-
-    def __repr__(self):
-        return '<%s (path=%s)>' % (self.__class__.__name__, self.path)

lib/apscheduler/jobstores/sqlalchemy.py

@@ -0,0 +1,137 @@
+from __future__ import absolute_import
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import maybe_ref, datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from sqlalchemy import create_engine, Table, Column, MetaData, Unicode, Float, LargeBinary, select
+    from sqlalchemy.exc import IntegrityError
+except ImportError:  # pragma: nocover
+    raise ImportError('SQLAlchemyJobStore requires SQLAlchemy installed')
+
+
+class SQLAlchemyJobStore(BaseJobStore):
+    """
+    Stores jobs in a database table using SQLAlchemy. The table will be created if it doesn't exist in the database.
+
+    Plugin alias: ``sqlalchemy``
+
+    :param str url: connection string (see `SQLAlchemy documentation
+                    <http://docs.sqlalchemy.org/en/latest/core/engines.html?highlight=create_engine#database-urls>`_
+                    on this)
+    :param engine: an SQLAlchemy Engine to use instead of creating a new one based on ``url``
+    :param str tablename: name of the table to store jobs in
+    :param metadata: a :class:`~sqlalchemy.MetaData` instance to use instead of creating a new one
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    """
+
+    def __init__(self, url=None, engine=None, tablename='apscheduler_jobs', metadata=None,
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL):
+        super(SQLAlchemyJobStore, self).__init__()
+        self.pickle_protocol = pickle_protocol
+        metadata = maybe_ref(metadata) or MetaData()
+
+        if engine:
+            self.engine = maybe_ref(engine)
+        elif url:
+            self.engine = create_engine(url)
+        else:
+            raise ValueError('Need either "engine" or "url" defined')
+
+        # 191 = max key length in MySQL for InnoDB/utf8mb4 tables, 25 = precision that translates to an 8-byte float
+        self.jobs_t = Table(
+            tablename, metadata,
+            Column('id', Unicode(191, _warn_on_bytestring=False), primary_key=True),
+            Column('next_run_time', Float(25), index=True),
+            Column('job_state', LargeBinary, nullable=False)
+        )
+
+        self.jobs_t.create(self.engine, True)
+
+    def lookup_job(self, job_id):
+        selectable = select([self.jobs_t.c.job_state]).where(self.jobs_t.c.id == job_id)
+        job_state = self.engine.execute(selectable).scalar()
+        return self._reconstitute_job(job_state) if job_state else None
+
+    def get_due_jobs(self, now):
+        timestamp = datetime_to_utc_timestamp(now)
+        return self._get_jobs(self.jobs_t.c.next_run_time <= timestamp)
+
+    def get_next_run_time(self):
+        selectable = select([self.jobs_t.c.next_run_time]).where(self.jobs_t.c.next_run_time != None).\
+            order_by(self.jobs_t.c.next_run_time).limit(1)
+        next_run_time = self.engine.execute(selectable).scalar()
+        return utc_timestamp_to_datetime(next_run_time)
+
+    def get_all_jobs(self):
+        return self._get_jobs()
+
+    def add_job(self, job):
+        insert = self.jobs_t.insert().values(**{
+            'id': job.id,
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': pickle.dumps(job.__getstate__(), self.pickle_protocol)
+        })
+        try:
+            self.engine.execute(insert)
+        except IntegrityError:
+            raise ConflictingIdError(job.id)
+
+    def update_job(self, job):
+        update = self.jobs_t.update().values(**{
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': pickle.dumps(job.__getstate__(), self.pickle_protocol)
+        }).where(self.jobs_t.c.id == job.id)
+        result = self.engine.execute(update)
+        if result.rowcount == 0:
+            raise JobLookupError(id)
+
+    def remove_job(self, job_id):
+        delete = self.jobs_t.delete().where(self.jobs_t.c.id == job_id)
+        result = self.engine.execute(delete)
+        if result.rowcount == 0:
+            raise JobLookupError(job_id)
+
+    def remove_all_jobs(self):
+        delete = self.jobs_t.delete()
+        self.engine.execute(delete)
+
+    def shutdown(self):
+        self.engine.dispose()
+
+    def _reconstitute_job(self, job_state):
+        job_state = pickle.loads(job_state)
+        job_state['jobstore'] = self
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _get_jobs(self, *conditions):
+        jobs = []
+        selectable = select([self.jobs_t.c.id, self.jobs_t.c.job_state]).order_by(self.jobs_t.c.next_run_time)
+        selectable = selectable.where(*conditions) if conditions else selectable
+        failed_job_ids = set()
+        for row in self.engine.execute(selectable):
+            try:
+                jobs.append(self._reconstitute_job(row.job_state))
+            except:
+                self._logger.exception('Unable to restore job "%s" -- removing it', row.id)
+                failed_job_ids.add(row.id)
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            delete = self.jobs_t.delete().where(self.jobs_t.c.id.in_(failed_job_ids))
+            self.engine.execute(delete)
+
+        return jobs
+
+    def __repr__(self):
+        return '<%s (url=%s)>' % (self.__class__.__name__, self.engine.url)

lib/apscheduler/jobstores/sqlalchemy_store.py

@@ -1,87 +0,0 @@
-"""
-Stores jobs in a database table using SQLAlchemy.
-"""
-import pickle
-import logging
-
-from apscheduler.jobstores.base import JobStore
-from apscheduler.job import Job
-
-try:
-    from sqlalchemy import *
-except ImportError:  # pragma: nocover
-    raise ImportError('SQLAlchemyJobStore requires SQLAlchemy installed')
-
-logger = logging.getLogger(__name__)
-
-
-class SQLAlchemyJobStore(JobStore):
-    def __init__(self, url=None, engine=None, tablename='apscheduler_jobs',
-                 metadata=None, pickle_protocol=pickle.HIGHEST_PROTOCOL):
-        self.jobs = []
-        self.pickle_protocol = pickle_protocol
-
-        if engine:
-            self.engine = engine
-        elif url:
-            self.engine = create_engine(url)
-        else:
-            raise ValueError('Need either "engine" or "url" defined')
-
-        self.jobs_t = Table(tablename, metadata or MetaData(),
-            Column('id', Integer,
-                   Sequence(tablename + '_id_seq', optional=True),
-                   primary_key=True),
-            Column('trigger', PickleType(pickle_protocol, mutable=False),
-                   nullable=False),
-            Column('func_ref', String(1024), nullable=False),
-            Column('args', PickleType(pickle_protocol, mutable=False),
-                   nullable=False),
-            Column('kwargs', PickleType(pickle_protocol, mutable=False),
-                   nullable=False),
-            Column('name', Unicode(1024), unique=True),
-            Column('misfire_grace_time', Integer, nullable=False),
-            Column('coalesce', Boolean, nullable=False),
-            Column('max_runs', Integer),
-            Column('max_instances', Integer),
-            Column('next_run_time', DateTime, nullable=False),
-            Column('runs', BigInteger))
-
-        self.jobs_t.create(self.engine, True)
-
-    def add_job(self, job):
-        job_dict = job.__getstate__()
-        result = self.engine.execute(self.jobs_t.insert().values(**job_dict))
-        job.id = result.inserted_primary_key[0]
-        self.jobs.append(job)
-
-    def remove_job(self, job):
-        delete = self.jobs_t.delete().where(self.jobs_t.c.id == job.id)
-        self.engine.execute(delete)
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        jobs = []
-        for row in self.engine.execute(select([self.jobs_t])):
-            try:
-                job = Job.__new__(Job)
-                job_dict = dict(row.items())
-                job.__setstate__(job_dict)
-                jobs.append(job)
-            except Exception:
-                job_name = job_dict.get('name', '(unknown)')
-                logger.exception('Unable to restore job "%s"', job_name)
-        self.jobs = jobs
-
-    def update_job(self, job):
-        job_dict = job.__getstate__()
-        update = self.jobs_t.update().where(self.jobs_t.c.id == job.id).\
-            values(next_run_time=job_dict['next_run_time'],
-                   runs=job_dict['runs'])
-        self.engine.execute(update)
-
-    def close(self):
-        self.engine.dispose()
-
-    def __repr__(self):
-        return '<%s (url=%s)>' % (self.__class__.__name__, self.engine.url)

lib/apscheduler/scheduler.py

@@ -1,559 +0,0 @@
-"""
-This module is the main part of the library. It houses the Scheduler class
-and related exceptions.
-"""
-
-from threading import Thread, Event, Lock
-from datetime import datetime, timedelta
-from logging import getLogger
-import os
-import sys
-
-from apscheduler.util import *
-from apscheduler.triggers import SimpleTrigger, IntervalTrigger, CronTrigger
-from apscheduler.jobstores.ram_store import RAMJobStore
-from apscheduler.job import Job, MaxInstancesReachedError
-from apscheduler.events import *
-from apscheduler.threadpool import ThreadPool
-
-logger = getLogger(__name__)
-
-
-class SchedulerAlreadyRunningError(Exception):
-    """
-    Raised when attempting to start or configure the scheduler when it's
-    already running.
-    """
-
-    def __str__(self):
-        return 'Scheduler is already running'
-
-
-class Scheduler(object):
-    """
-    This class is responsible for scheduling jobs and triggering
-    their execution.
-    """
-
-    _stopped = False
-    _thread = None
-
-    def __init__(self, gconfig={}, **options):
-        self._wakeup = Event()
-        self._jobstores = {}
-        self._jobstores_lock = Lock()
-        self._listeners = []
-        self._listeners_lock = Lock()
-        self._pending_jobs = []
-        self.configure(gconfig, **options)
-
-    def configure(self, gconfig={}, **options):
-        """
-        Reconfigures the scheduler with the given options. Can only be done
-        when the scheduler isn't running.
-        """
-        if self.running:
-            raise SchedulerAlreadyRunningError
-
-        # Set general options
-        config = combine_opts(gconfig, 'apscheduler.', options)
-        self.misfire_grace_time = int(config.pop('misfire_grace_time', 1))
-        self.coalesce = asbool(config.pop('coalesce', True))
-        self.daemonic = asbool(config.pop('daemonic', True))
-
-        # Configure the thread pool
-        if 'threadpool' in config:
-            self._threadpool = maybe_ref(config['threadpool'])
-        else:
-            threadpool_opts = combine_opts(config, 'threadpool.')
-            self._threadpool = ThreadPool(**threadpool_opts)
-
-        # Configure job stores
-        jobstore_opts = combine_opts(config, 'jobstore.')
-        jobstores = {}
-        for key, value in jobstore_opts.items():
-            store_name, option = key.split('.', 1)
-            opts_dict = jobstores.setdefault(store_name, {})
-            opts_dict[option] = value
-
-        for alias, opts in jobstores.items():
-            classname = opts.pop('class')
-            cls = maybe_ref(classname)
-            jobstore = cls(**opts)
-            self.add_jobstore(jobstore, alias, True)
-
-    def start(self):
-        """
-        Starts the scheduler in a new thread.
-        """
-        if self.running:
-            raise SchedulerAlreadyRunningError
-
-        # Create a RAMJobStore as the default if there is no default job store
-        if not 'default' in self._jobstores:
-            self.add_jobstore(RAMJobStore(), 'default', True)
-
-        # Schedule all pending jobs
-        for job, jobstore in self._pending_jobs:
-            self._real_add_job(job, jobstore, False)
-        del self._pending_jobs[:]
-
-        self._stopped = False
-        self._thread = Thread(target=self._main_loop, name='APScheduler')
-        self._thread.setDaemon(self.daemonic)
-        self._thread.start()
-
-    def shutdown(self, wait=True, shutdown_threadpool=True):
-        """
-        Shuts down the scheduler and terminates the thread.
-        Does not interrupt any currently running jobs.
-
-        :param wait: ``True`` to wait until all currently executing jobs have
-                     finished (if ``shutdown_threadpool`` is also ``True``)
-        :param shutdown_threadpool: ``True`` to shut down the thread pool
-        """
-        if not self.running:
-            return
-
-        self._stopped = True
-        self._wakeup.set()
-
-        # Shut down the thread pool
-        if shutdown_threadpool:
-            self._threadpool.shutdown(wait)
-
-        # Wait until the scheduler thread terminates
-        self._thread.join()
-
-    @property
-    def running(self):
-        return not self._stopped and self._thread and self._thread.isAlive()
-
-    def add_jobstore(self, jobstore, alias, quiet=False):
-        """
-        Adds a job store to this scheduler.
-
-        :param jobstore: job store to be added
-        :param alias: alias for the job store
-        :param quiet: True to suppress scheduler thread wakeup
-        :type jobstore: instance of
-            :class:`~apscheduler.jobstores.base.JobStore`
-        :type alias: str
-        """
-        self._jobstores_lock.acquire()
-        try:
-            if alias in self._jobstores:
-                raise KeyError('Alias "%s" is already in use' % alias)
-            self._jobstores[alias] = jobstore
-            jobstore.load_jobs()
-        finally:
-            self._jobstores_lock.release()
-
-        # Notify listeners that a new job store has been added
-        self._notify_listeners(JobStoreEvent(EVENT_JOBSTORE_ADDED, alias))
-
-        # Notify the scheduler so it can scan the new job store for jobs
-        if not quiet:
-            self._wakeup.set()
-
-    def remove_jobstore(self, alias):
-        """
-        Removes the job store by the given alias from this scheduler.
-
-        :type alias: str
-        """
-        self._jobstores_lock.acquire()
-        try:
-            try:
-                del self._jobstores[alias]
-            except KeyError:
-                raise KeyError('No such job store: %s' % alias)
-        finally:
-            self._jobstores_lock.release()
-
-        # Notify listeners that a job store has been removed
-        self._notify_listeners(JobStoreEvent(EVENT_JOBSTORE_REMOVED, alias))
-
-    def add_listener(self, callback, mask=EVENT_ALL):
-        """
-        Adds a listener for scheduler events. When a matching event occurs,
-        ``callback`` is executed with the event object as its sole argument.
-        If the ``mask`` parameter is not provided, the callback will receive
-        events of all types.
-
-        :param callback: any callable that takes one argument
-        :param mask: bitmask that indicates which events should be listened to
-        """
-        self._listeners_lock.acquire()
-        try:
-            self._listeners.append((callback, mask))
-        finally:
-            self._listeners_lock.release()
-
-    def remove_listener(self, callback):
-        """
-        Removes a previously added event listener.
-        """
-        self._listeners_lock.acquire()
-        try:
-            for i, (cb, _) in enumerate(self._listeners):
-                if callback == cb:
-                    del self._listeners[i]
-        finally:
-            self._listeners_lock.release()
-
-    def _notify_listeners(self, event):
-        self._listeners_lock.acquire()
-        try:
-            listeners = tuple(self._listeners)
-        finally:
-            self._listeners_lock.release()
-
-        for cb, mask in listeners:
-            if event.code & mask:
-                try:
-                    cb(event)
-                except:
-                    logger.exception('Error notifying listener')
-
-    def _real_add_job(self, job, jobstore, wakeup):
-        job.compute_next_run_time(datetime.now())
-        if not job.next_run_time:
-            raise ValueError('Not adding job since it would never be run')
-
-        self._jobstores_lock.acquire()
-        try:
-            try:
-                store = self._jobstores[jobstore]
-            except KeyError:
-                raise KeyError('No such job store: %s' % jobstore)
-            store.add_job(job)
-        finally:
-            self._jobstores_lock.release()
-
-        # Notify listeners that a new job has been added
-        event = JobStoreEvent(EVENT_JOBSTORE_JOB_ADDED, jobstore, job)
-        self._notify_listeners(event)
-
-        logger.info('Added job "%s" to job store "%s"', job, jobstore)
-
-        # Notify the scheduler about the new job
-        if wakeup:
-            self._wakeup.set()
-
-    def add_job(self, trigger, func, args, kwargs, jobstore='default',
-                **options):
-        """
-        Adds the given job to the job list and notifies the scheduler thread.
-
-        :param trigger: alias of the job store to store the job in
-        :param func: callable to run at the given time
-        :param args: list of positional arguments to call func with
-        :param kwargs: dict of keyword arguments to call func with
-        :param jobstore: alias of the job store to store the job in
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        job = Job(trigger, func, args or [], kwargs or {},
-                  options.pop('misfire_grace_time', self.misfire_grace_time),
-                  options.pop('coalesce', self.coalesce), **options)
-        if not self.running:
-            self._pending_jobs.append((job, jobstore))
-            logger.info('Adding job tentatively -- it will be properly '
-                        'scheduled when the scheduler starts')
-        else:
-            self._real_add_job(job, jobstore, True)
-        return job
-
-    def _remove_job(self, job, alias, jobstore):
-        jobstore.remove_job(job)
-
-        # Notify listeners that a job has been removed
-        event = JobStoreEvent(EVENT_JOBSTORE_JOB_REMOVED, alias, job)
-        self._notify_listeners(event)
-
-        logger.info('Removed job "%s"', job)
-
-    def add_date_job(self, func, date, args=None, kwargs=None, **options):
-        """
-        Schedules a job to be completed on a specific date and time.
-
-        :param func: callable to run at the given time
-        :param date: the date/time to run the job at
-        :param name: name of the job
-        :param jobstore: stored the job in the named (or given) job store
-        :param misfire_grace_time: seconds after the designated run time that
-            the job is still allowed to be run
-        :type date: :class:`datetime.date`
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        trigger = SimpleTrigger(date)
-        return self.add_job(trigger, func, args, kwargs, **options)
-
-    def add_interval_job(self, func, weeks=0, days=0, hours=0, minutes=0,
-                         seconds=0, start_date=None, args=None, kwargs=None,
-                         **options):
-        """
-        Schedules a job to be completed on specified intervals.
-
-        :param func: callable to run
-        :param weeks: number of weeks to wait
-        :param days: number of days to wait
-        :param hours: number of hours to wait
-        :param minutes: number of minutes to wait
-        :param seconds: number of seconds to wait
-        :param start_date: when to first execute the job and start the
-            counter (default is after the given interval)
-        :param args: list of positional arguments to call func with
-        :param kwargs: dict of keyword arguments to call func with
-        :param name: name of the job
-        :param jobstore: alias of the job store to add the job to
-        :param misfire_grace_time: seconds after the designated run time that
-            the job is still allowed to be run
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        interval = timedelta(weeks=weeks, days=days, hours=hours,
-                             minutes=minutes, seconds=seconds)
-        trigger = IntervalTrigger(interval, start_date)
-        return self.add_job(trigger, func, args, kwargs, **options)
-
-    def add_cron_job(self, func, year='*', month='*', day='*', week='*',
-                     day_of_week='*', hour='*', minute='*', second='*',
-                     start_date=None, args=None, kwargs=None, **options):
-        """
-        Schedules a job to be completed on times that match the given
-        expressions.
-
-        :param func: callable to run
-        :param year: year to run on
-        :param month: month to run on (0 = January)
-        :param day: day of month to run on
-        :param week: week of the year to run on
-        :param day_of_week: weekday to run on (0 = Monday)
-        :param hour: hour to run on
-        :param second: second to run on
-        :param args: list of positional arguments to call func with
-        :param kwargs: dict of keyword arguments to call func with
-        :param name: name of the job
-        :param jobstore: alias of the job store to add the job to
-        :param misfire_grace_time: seconds after the designated run time that
-            the job is still allowed to be run
-        :return: the scheduled job
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        trigger = CronTrigger(year=year, month=month, day=day, week=week,
-                              day_of_week=day_of_week, hour=hour,
-                              minute=minute, second=second,
-                              start_date=start_date)
-        return self.add_job(trigger, func, args, kwargs, **options)
-
-    def cron_schedule(self, **options):
-        """
-        Decorator version of :meth:`add_cron_job`.
-        This decorator does not wrap its host function.
-        Unscheduling decorated functions is possible by passing the ``job``
-        attribute of the scheduled function to :meth:`unschedule_job`.
-        """
-        def inner(func):
-            func.job = self.add_cron_job(func, **options)
-            return func
-        return inner
-
-    def interval_schedule(self, **options):
-        """
-        Decorator version of :meth:`add_interval_job`.
-        This decorator does not wrap its host function.
-        Unscheduling decorated functions is possible by passing the ``job``
-        attribute of the scheduled function to :meth:`unschedule_job`.
-        """
-        def inner(func):
-            func.job = self.add_interval_job(func, **options)
-            return func
-        return inner
-
-    def get_jobs(self):
-        """
-        Returns a list of all scheduled jobs.
-
-        :return: list of :class:`~apscheduler.job.Job` objects
-        """
-        self._jobstores_lock.acquire()
-        try:
-            jobs = []
-            for jobstore in itervalues(self._jobstores):
-                jobs.extend(jobstore.jobs)
-            return jobs
-        finally:
-            self._jobstores_lock.release()
-
-    def unschedule_job(self, job):
-        """
-        Removes a job, preventing it from being run any more.
-        """
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                if job in list(jobstore.jobs):
-                    self._remove_job(job, alias, jobstore)
-                    return
-        finally:
-            self._jobstores_lock.release()
-
-        raise KeyError('Job "%s" is not scheduled in any job store' % job)
-
-    def unschedule_func(self, func):
-        """
-        Removes all jobs that would execute the given function.
-        """
-        found = False
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                for job in list(jobstore.jobs):
-                    if job.func == func:
-                        self._remove_job(job, alias, jobstore)
-                        found = True
-        finally:
-            self._jobstores_lock.release()
-
-        if not found:
-            raise KeyError('The given function is not scheduled in this '
-                           'scheduler')
-
-    def print_jobs(self, out=None):
-        """
-        Prints out a textual listing of all jobs currently scheduled on this
-        scheduler.
-
-        :param out: a file-like object to print to (defaults to **sys.stdout**
-                    if nothing is given)
-        """
-        out = out or sys.stdout
-        job_strs = []
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                job_strs.append('Jobstore %s:' % alias)
-                if jobstore.jobs:
-                    for job in jobstore.jobs:
-                        job_strs.append('    %s' % job)
-                else:
-                    job_strs.append('    No scheduled jobs')
-        finally:
-            self._jobstores_lock.release()
-
-        out.write(os.linesep.join(job_strs))
-
-    def _run_job(self, job, run_times):
-        """
-        Acts as a harness that runs the actual job code in a thread.
-        """
-        for run_time in run_times:
-            # See if the job missed its run time window, and handle possible
-            # misfires accordingly
-            difference = datetime.now() - run_time
-            grace_time = timedelta(seconds=job.misfire_grace_time)
-            if difference > grace_time:
-                # Notify listeners about a missed run
-                event = JobEvent(EVENT_JOB_MISSED, job, run_time)
-                self._notify_listeners(event)
-                logger.warning('Run time of job "%s" was missed by %s',
-                               job, difference)
-            else:
-                try:
-                    job.add_instance()
-                except MaxInstancesReachedError:
-                    event = JobEvent(EVENT_JOB_MISSED, job, run_time)
-                    self._notify_listeners(event)
-                    logger.warning('Execution of job "%s" skipped: '
-                                   'maximum number of running instances '
-                                   'reached (%d)', job, job.max_instances)
-                    break
-
-                logger.info('Running job "%s" (scheduled at %s)', job,
-                            run_time)
-
-                try:
-                    retval = job.func(*job.args, **job.kwargs)
-                except:
-                    # Notify listeners about the exception
-                    exc, tb = sys.exc_info()[1:]
-                    event = JobEvent(EVENT_JOB_ERROR, job, run_time,
-                                     exception=exc, traceback=tb)
-                    self._notify_listeners(event)
-
-                    logger.exception('Job "%s" raised an exception', job)
-                else:
-                    # Notify listeners about successful execution
-                    event = JobEvent(EVENT_JOB_EXECUTED, job, run_time,
-                                     retval=retval)
-                    self._notify_listeners(event)
-
-                    logger.info('Job "%s" executed successfully', job)
-
-                job.remove_instance()
-
-                # If coalescing is enabled, don't attempt any further runs
-                if job.coalesce:
-                    break
-
-    def _process_jobs(self, now):
-        """
-        Iterates through jobs in every jobstore, starts pending jobs
-        and figures out the next wakeup time.
-        """
-        next_wakeup_time = None
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                for job in tuple(jobstore.jobs):
-                    run_times = job.get_run_times(now)
-                    if run_times:
-                        self._threadpool.submit(self._run_job, job, run_times)
-
-                        # Increase the job's run count
-                        if job.coalesce:
-                            job.runs += 1
-                        else:
-                            job.runs += len(run_times)
-
-                        # Update the job, but don't keep finished jobs around
-                        if job.compute_next_run_time(now + timedelta(microseconds=1)):
-                            jobstore.update_job(job)
-                        else:
-                            self._remove_job(job, alias, jobstore)
-
-                    if not next_wakeup_time:
-                        next_wakeup_time = job.next_run_time
-                    elif job.next_run_time:
-                        next_wakeup_time = min(next_wakeup_time,
-                                               job.next_run_time)
-            return next_wakeup_time
-        finally:
-            self._jobstores_lock.release()
-
-    def _main_loop(self):
-        """Executes jobs on schedule."""
-
-        logger.info('Scheduler started')
-        self._notify_listeners(SchedulerEvent(EVENT_SCHEDULER_START))
-
-        self._wakeup.clear()
-        while not self._stopped:
-            logger.debug('Looking for jobs to run')
-            now = datetime.now()
-            next_wakeup_time = self._process_jobs(now)
-
-            # Sleep until the next job is scheduled to be run,
-            # a new job is added or the scheduler is stopped
-            if next_wakeup_time is not None:
-                wait_seconds = time_difference(next_wakeup_time, now)
-                logger.debug('Next wakeup is due at %s (in %f seconds)',
-                             next_wakeup_time, wait_seconds)
-                self._wakeup.wait(wait_seconds)
-            else:
-                logger.debug('No jobs; waiting until a job is added')
-                self._wakeup.wait()
-            self._wakeup.clear()
-
-        logger.info('Scheduler has been shut down')
-        self._notify_listeners(SchedulerEvent(EVENT_SCHEDULER_SHUTDOWN))

lib/apscheduler/schedulers/__init__.py

@@ -0,0 +1,12 @@
+class SchedulerAlreadyRunningError(Exception):
+    """Raised when attempting to start or configure the scheduler when it's already running."""
+
+    def __str__(self):
+        return 'Scheduler is already running'
+
+
+class SchedulerNotRunningError(Exception):
+    """Raised when attempting to shutdown the scheduler when it's not running."""
+
+    def __str__(self):
+        return 'Scheduler is not running'

lib/apscheduler/schedulers/asyncio.py

@@ -0,0 +1,68 @@
+from __future__ import absolute_import
+from functools import wraps
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.util import maybe_ref
+
+try:
+    import asyncio
+except ImportError:  # pragma: nocover
+    try:
+        import trollius as asyncio
+    except ImportError:
+        raise ImportError('AsyncIOScheduler requires either Python 3.4 or the asyncio package installed')
+
+
+def run_in_event_loop(func):
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        self._eventloop.call_soon_threadsafe(func, self, *args, **kwargs)
+    return wrapper
+
+
+class AsyncIOScheduler(BaseScheduler):
+    """
+    A scheduler that runs on an asyncio (:pep:`3156`) event loop.
+
+    Extra options:
+
+    ============== =============================================================
+    ``event_loop`` AsyncIO event loop to use (defaults to the global event loop)
+    ============== =============================================================
+    """
+
+    _eventloop = None
+    _timeout = None
+
+    def start(self):
+        super(AsyncIOScheduler, self).start()
+        self.wakeup()
+
+    @run_in_event_loop
+    def shutdown(self, wait=True):
+        super(AsyncIOScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _configure(self, config):
+        self._eventloop = maybe_ref(config.pop('event_loop', None)) or asyncio.get_event_loop()
+        super(AsyncIOScheduler, self)._configure(config)
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._timeout = self._eventloop.call_later(wait_seconds, self.wakeup)
+
+    def _stop_timer(self):
+        if self._timeout:
+            self._timeout.cancel()
+            del self._timeout
+
+    @run_in_event_loop
+    def wakeup(self):
+        self._stop_timer()
+        wait_seconds = self._process_jobs()
+        self._start_timer(wait_seconds)
+
+    def _create_default_executor(self):
+        from apscheduler.executors.asyncio import AsyncIOExecutor
+        return AsyncIOExecutor()

lib/apscheduler/schedulers/background.py

@@ -0,0 +1,39 @@
+from __future__ import absolute_import
+from threading import Thread, Event
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.schedulers.blocking import BlockingScheduler
+from apscheduler.util import asbool
+
+
+class BackgroundScheduler(BlockingScheduler):
+    """
+    A scheduler that runs in the background using a separate thread
+    (:meth:`~apscheduler.schedulers.base.BaseScheduler.start` will return immediately).
+
+    Extra options:
+
+    ========== ============================================================================================
+    ``daemon`` Set the ``daemon`` option in the background thread (defaults to ``True``,
+               see `the documentation <https://docs.python.org/3.4/library/threading.html#thread-objects>`_
+               for further details)
+    ========== ============================================================================================
+    """
+
+    _thread = None
+
+    def _configure(self, config):
+        self._daemon = asbool(config.pop('daemon', True))
+        super(BackgroundScheduler, self)._configure(config)
+
+    def start(self):
+        BaseScheduler.start(self)
+        self._event = Event()
+        self._thread = Thread(target=self._main_loop, name='APScheduler')
+        self._thread.daemon = self._daemon
+        self._thread.start()
+
+    def shutdown(self, wait=True):
+        super(BackgroundScheduler, self).shutdown(wait)
+        self._thread.join()
+        del self._thread

lib/apscheduler/schedulers/base.py

@@ -0,0 +1,845 @@
+from __future__ import print_function
+from abc import ABCMeta, abstractmethod
+from collections import MutableMapping
+from threading import RLock
+from datetime import datetime
+from logging import getLogger
+import sys
+
+from pkg_resources import iter_entry_points
+from tzlocal import get_localzone
+import six
+
+from apscheduler.schedulers import SchedulerAlreadyRunningError, SchedulerNotRunningError
+from apscheduler.executors.base import MaxInstancesReachedError, BaseExecutor
+from apscheduler.executors.pool import ThreadPoolExecutor
+from apscheduler.jobstores.base import ConflictingIdError, JobLookupError, BaseJobStore
+from apscheduler.jobstores.memory import MemoryJobStore
+from apscheduler.job import Job
+from apscheduler.triggers.base import BaseTrigger
+from apscheduler.util import asbool, asint, astimezone, maybe_ref, timedelta_seconds, undefined
+from apscheduler.events import (
+    SchedulerEvent, JobEvent, EVENT_SCHEDULER_START, EVENT_SCHEDULER_SHUTDOWN, EVENT_JOBSTORE_ADDED,
+    EVENT_JOBSTORE_REMOVED, EVENT_ALL, EVENT_JOB_MODIFIED, EVENT_JOB_REMOVED, EVENT_JOB_ADDED, EVENT_EXECUTOR_ADDED,
+    EVENT_EXECUTOR_REMOVED, EVENT_ALL_JOBS_REMOVED)
+
+
+class BaseScheduler(six.with_metaclass(ABCMeta)):
+    """
+    Abstract base class for all schedulers. Takes the following keyword arguments:
+
+    :param str|logging.Logger logger: logger to use for the scheduler's logging (defaults to apscheduler.scheduler)
+    :param str|datetime.tzinfo timezone: the default time zone (defaults to the local timezone)
+    :param dict job_defaults: default values for newly added jobs
+    :param dict jobstores: a dictionary of job store alias -> job store instance or configuration dict
+    :param dict executors: a dictionary of executor alias -> executor instance or configuration dict
+
+    .. seealso:: :ref:`scheduler-config`
+    """
+
+    _trigger_plugins = dict((ep.name, ep) for ep in iter_entry_points('apscheduler.triggers'))
+    _trigger_classes = {}
+    _executor_plugins = dict((ep.name, ep) for ep in iter_entry_points('apscheduler.executors'))
+    _executor_classes = {}
+    _jobstore_plugins = dict((ep.name, ep) for ep in iter_entry_points('apscheduler.jobstores'))
+    _jobstore_classes = {}
+    _stopped = True
+
+    #
+    # Public API
+    #
+
+    def __init__(self, gconfig={}, **options):
+        super(BaseScheduler, self).__init__()
+        self._executors = {}
+        self._executors_lock = self._create_lock()
+        self._jobstores = {}
+        self._jobstores_lock = self._create_lock()
+        self._listeners = []
+        self._listeners_lock = self._create_lock()
+        self._pending_jobs = []
+        self.configure(gconfig, **options)
+
+    def configure(self, gconfig={}, prefix='apscheduler.', **options):
+        """
+        Reconfigures the scheduler with the given options. Can only be done when the scheduler isn't running.
+
+        :param dict gconfig: a "global" configuration dictionary whose values can be overridden by keyword arguments to
+                             this method
+        :param str|unicode prefix: pick only those keys from ``gconfig`` that are prefixed with this string
+                                   (pass an empty string or ``None`` to use all keys)
+        :raises SchedulerAlreadyRunningError: if the scheduler is already running
+        """
+
+        if self.running:
+            raise SchedulerAlreadyRunningError
+
+        # If a non-empty prefix was given, strip it from the keys in the global configuration dict
+        if prefix:
+            prefixlen = len(prefix)
+            gconfig = dict((key[prefixlen:], value) for key, value in six.iteritems(gconfig) if key.startswith(prefix))
+
+        # Create a structure from the dotted options (e.g. "a.b.c = d" -> {'a': {'b': {'c': 'd'}}})
+        config = {}
+        for key, value in six.iteritems(gconfig):
+            parts = key.split('.')
+            parent = config
+            key = parts.pop(0)
+            while parts:
+                parent = parent.setdefault(key, {})
+                key = parts.pop(0)
+            parent[key] = value
+
+        # Override any options with explicit keyword arguments
+        config.update(options)
+        self._configure(config)
+
+    @abstractmethod
+    def start(self):
+        """
+        Starts the scheduler. The details of this process depend on the implementation.
+
+        :raises SchedulerAlreadyRunningError: if the scheduler is already running
+        """
+
+        if self.running:
+            raise SchedulerAlreadyRunningError
+
+        with self._executors_lock:
+            # Create a default executor if nothing else is configured
+            if 'default' not in self._executors:
+                self.add_executor(self._create_default_executor(), 'default')
+
+            # Start all the executors
+            for alias, executor in six.iteritems(self._executors):
+                executor.start(self, alias)
+
+        with self._jobstores_lock:
+            # Create a default job store if nothing else is configured
+            if 'default' not in self._jobstores:
+                self.add_jobstore(self._create_default_jobstore(), 'default')
+
+            # Start all the job stores
+            for alias, store in six.iteritems(self._jobstores):
+                store.start(self, alias)
+
+            # Schedule all pending jobs
+            for job, jobstore_alias, replace_existing in self._pending_jobs:
+                self._real_add_job(job, jobstore_alias, replace_existing, False)
+            del self._pending_jobs[:]
+
+        self._stopped = False
+        self._logger.info('Scheduler started')
+
+        # Notify listeners that the scheduler has been started
+        self._dispatch_event(SchedulerEvent(EVENT_SCHEDULER_START))
+
+    @abstractmethod
+    def shutdown(self, wait=True):
+        """
+        Shuts down the scheduler. Does not interrupt any currently running jobs.
+
+        :param bool wait: ``True`` to wait until all currently executing jobs have finished
+        :raises SchedulerNotRunningError: if the scheduler has not been started yet
+        """
+
+        if not self.running:
+            raise SchedulerNotRunningError
+
+        self._stopped = True
+
+        # Shut down all executors
+        for executor in six.itervalues(self._executors):
+            executor.shutdown(wait)
+
+        # Shut down all job stores
+        for jobstore in six.itervalues(self._jobstores):
+            jobstore.shutdown()
+
+        self._logger.info('Scheduler has been shut down')
+        self._dispatch_event(SchedulerEvent(EVENT_SCHEDULER_SHUTDOWN))
+
+    @property
+    def running(self):
+        return not self._stopped
+
+    def add_executor(self, executor, alias='default', **executor_opts):
+        """
+        Adds an executor to this scheduler. Any extra keyword arguments will be passed to the executor plugin's
+        constructor, assuming that the first argument is the name of an executor plugin.
+
+        :param str|unicode|apscheduler.executors.base.BaseExecutor executor: either an executor instance or the name of
+            an executor plugin
+        :param str|unicode alias: alias for the scheduler
+        :raises ValueError: if there is already an executor by the given alias
+        """
+
+        with self._executors_lock:
+            if alias in self._executors:
+                raise ValueError('This scheduler already has an executor by the alias of "%s"' % alias)
+
+            if isinstance(executor, BaseExecutor):
+                self._executors[alias] = executor
+            elif isinstance(executor, six.string_types):
+                self._executors[alias] = executor = self._create_plugin_instance('executor', executor, executor_opts)
+            else:
+                raise TypeError('Expected an executor instance or a string, got %s instead' %
+                                executor.__class__.__name__)
+
+            # Start the executor right away if the scheduler is running
+            if self.running:
+                executor.start(self)
+
+        self._dispatch_event(SchedulerEvent(EVENT_EXECUTOR_ADDED, alias))
+
+    def remove_executor(self, alias, shutdown=True):
+        """
+        Removes the executor by the given alias from this scheduler.
+
+        :param str|unicode alias: alias of the executor
+        :param bool shutdown: ``True`` to shut down the executor after removing it
+        """
+
+        with self._jobstores_lock:
+            executor = self._lookup_executor(alias)
+            del self._executors[alias]
+
+        if shutdown:
+            executor.shutdown()
+
+        self._dispatch_event(SchedulerEvent(EVENT_EXECUTOR_REMOVED, alias))
+
+    def add_jobstore(self, jobstore, alias='default', **jobstore_opts):
+        """
+        Adds a job store to this scheduler. Any extra keyword arguments will be passed to the job store plugin's
+        constructor, assuming that the first argument is the name of a job store plugin.
+
+        :param str|unicode|apscheduler.jobstores.base.BaseJobStore jobstore: job store to be added
+        :param str|unicode alias: alias for the job store
+        :raises ValueError: if there is already a job store by the given alias
+        """
+
+        with self._jobstores_lock:
+            if alias in self._jobstores:
+                raise ValueError('This scheduler already has a job store by the alias of "%s"' % alias)
+
+            if isinstance(jobstore, BaseJobStore):
+                self._jobstores[alias] = jobstore
+            elif isinstance(jobstore, six.string_types):
+                self._jobstores[alias] = jobstore = self._create_plugin_instance('jobstore', jobstore, jobstore_opts)
+            else:
+                raise TypeError('Expected a job store instance or a string, got %s instead' %
+                                jobstore.__class__.__name__)
+
+            # Start the job store right away if the scheduler is running
+            if self.running:
+                jobstore.start(self, alias)
+
+        # Notify listeners that a new job store has been added
+        self._dispatch_event(SchedulerEvent(EVENT_JOBSTORE_ADDED, alias))
+
+        # Notify the scheduler so it can scan the new job store for jobs
+        if self.running:
+            self.wakeup()
+
+    def remove_jobstore(self, alias, shutdown=True):
+        """
+        Removes the job store by the given alias from this scheduler.
+
+        :param str|unicode alias: alias of the job store
+        :param bool shutdown: ``True`` to shut down the job store after removing it
+        """
+
+        with self._jobstores_lock:
+            jobstore = self._lookup_jobstore(alias)
+            del self._jobstores[alias]
+
+        if shutdown:
+            jobstore.shutdown()
+
+        self._dispatch_event(SchedulerEvent(EVENT_JOBSTORE_REMOVED, alias))
+
+    def add_listener(self, callback, mask=EVENT_ALL):
+        """
+        add_listener(callback, mask=EVENT_ALL)
+
+        Adds a listener for scheduler events. When a matching event occurs, ``callback`` is executed with the event
+        object as its sole argument. If the ``mask`` parameter is not provided, the callback will receive events of all
+        types.
+
+        :param callback: any callable that takes one argument
+        :param int mask: bitmask that indicates which events should be listened to
+
+        .. seealso:: :mod:`apscheduler.events`
+        .. seealso:: :ref:`scheduler-events`
+        """
+
+        with self._listeners_lock:
+            self._listeners.append((callback, mask))
+
+    def remove_listener(self, callback):
+        """Removes a previously added event listener."""
+
+        with self._listeners_lock:
+            for i, (cb, _) in enumerate(self._listeners):
+                if callback == cb:
+                    del self._listeners[i]
+
+    def add_job(self, func, trigger=None, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined,
+                coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default',
+                executor='default', replace_existing=False, **trigger_args):
+        """
+        add_job(func, trigger=None, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined, \
+            coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default', \
+            executor='default', replace_existing=False, **trigger_args)
+
+        Adds the given job to the job list and wakes up the scheduler if it's already running.
+
+        Any option that defaults to ``undefined`` will be replaced with the corresponding default value when the job is
+        scheduled (which happens when the scheduler is started, or immediately if the scheduler is already running).
+
+        The ``func`` argument can be given either as a callable object or a textual reference in the
+        ``package.module:some.object`` format, where the first half (separated by ``:``) is an importable module and the
+        second half is a reference to the callable object, relative to the module.
+
+        The ``trigger`` argument can either be:
+          #. the alias name of the trigger (e.g. ``date``, ``interval`` or ``cron``), in which case any extra keyword
+             arguments to this method are passed on to the trigger's constructor
+          #. an instance of a trigger class
+
+        :param func: callable (or a textual reference to one) to run at the given time
+        :param str|apscheduler.triggers.base.BaseTrigger trigger: trigger that determines when ``func`` is called
+        :param list|tuple args: list of positional arguments to call func with
+        :param dict kwargs: dict of keyword arguments to call func with
+        :param str|unicode id: explicit identifier for the job (for modifying it later)
+        :param str|unicode name: textual description of the job
+        :param int misfire_grace_time: seconds after the designated run time that the job is still allowed to be run
+        :param bool coalesce: run once instead of many times if the scheduler determines that the job should be run more
+                              than once in succession
+        :param int max_instances: maximum number of concurrently running instances allowed for this job
+        :param datetime next_run_time: when to first run the job, regardless of the trigger (pass ``None`` to add the
+                                       job as paused)
+        :param str|unicode jobstore: alias of the job store to store the job in
+        :param str|unicode executor: alias of the executor to run the job with
+        :param bool replace_existing: ``True`` to replace an existing job with the same ``id`` (but retain the
+                                      number of runs from the existing one)
+        :rtype: Job
+        """
+
+        job_kwargs = {
+            'trigger': self._create_trigger(trigger, trigger_args),
+            'executor': executor,
+            'func': func,
+            'args': tuple(args) if args is not None else (),
+            'kwargs': dict(kwargs) if kwargs is not None else {},
+            'id': id,
+            'name': name,
+            'misfire_grace_time': misfire_grace_time,
+            'coalesce': coalesce,
+            'max_instances': max_instances,
+            'next_run_time': next_run_time
+        }
+        job_kwargs = dict((key, value) for key, value in six.iteritems(job_kwargs) if value is not undefined)
+        job = Job(self, **job_kwargs)
+
+        # Don't really add jobs to job stores before the scheduler is up and running
+        with self._jobstores_lock:
+            if not self.running:
+                self._pending_jobs.append((job, jobstore, replace_existing))
+                self._logger.info('Adding job tentatively -- it will be properly scheduled when the scheduler starts')
+            else:
+                self._real_add_job(job, jobstore, replace_existing, True)
+
+        return job
+
+    def scheduled_job(self, trigger, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined,
+                      coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default',
+                      executor='default', **trigger_args):
+        """
+        scheduled_job(trigger, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined, \
+            coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default', \
+            executor='default',**trigger_args)
+
+        A decorator version of :meth:`add_job`, except that ``replace_existing`` is always ``True``.
+
+        .. important:: The ``id`` argument must be given if scheduling a job in a persistent job store. The scheduler
+           cannot, however, enforce this requirement.
+        """
+
+        def inner(func):
+            self.add_job(func, trigger, args, kwargs, id, name, misfire_grace_time, coalesce, max_instances,
+                         next_run_time, jobstore, executor, True, **trigger_args)
+            return func
+        return inner
+
+    def modify_job(self, job_id, jobstore=None, **changes):
+        """
+        Modifies the properties of a single job. Modifications are passed to this method as extra keyword arguments.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        """
+        with self._jobstores_lock:
+            job, jobstore = self._lookup_job(job_id, jobstore)
+            job._modify(**changes)
+            if jobstore:
+                self._lookup_jobstore(jobstore).update_job(job)
+
+        self._dispatch_event(JobEvent(EVENT_JOB_MODIFIED, job_id, jobstore))
+
+        # Wake up the scheduler since the job's next run time may have been changed
+        self.wakeup()
+
+    def reschedule_job(self, job_id, jobstore=None, trigger=None, **trigger_args):
+        """
+        Constructs a new trigger for a job and updates its next run time.
+        Extra keyword arguments are passed directly to the trigger's constructor.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        :param trigger: alias of the trigger type or a trigger instance
+        """
+
+        trigger = self._create_trigger(trigger, trigger_args)
+        now = datetime.now(self.timezone)
+        next_run_time = trigger.get_next_fire_time(None, now)
+        self.modify_job(job_id, jobstore, trigger=trigger, next_run_time=next_run_time)
+
+    def pause_job(self, job_id, jobstore=None):
+        """
+        Causes the given job not to be executed until it is explicitly resumed.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        """
+
+        self.modify_job(job_id, jobstore, next_run_time=None)
+
+    def resume_job(self, job_id, jobstore=None):
+        """
+        Resumes the schedule of the given job, or removes the job if its schedule is finished.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        """
+
+        with self._jobstores_lock:
+            job, jobstore = self._lookup_job(job_id, jobstore)
+            now = datetime.now(self.timezone)
+            next_run_time = job.trigger.get_next_fire_time(None, now)
+            if next_run_time:
+                self.modify_job(job_id, jobstore, next_run_time=next_run_time)
+            else:
+                self.remove_job(job.id, jobstore)
+
+    def get_jobs(self, jobstore=None, pending=None):
+        """
+        Returns a list of pending jobs (if the scheduler hasn't been started yet) and scheduled jobs, either from a
+        specific job store or from all of them.
+
+        :param str|unicode jobstore: alias of the job store
+        :param bool pending: ``False`` to leave out pending jobs (jobs that are waiting for the scheduler start to be
+                             added to their respective job stores), ``True`` to only include pending jobs, anything else
+                             to return both
+        :rtype: list[Job]
+        """
+
+        with self._jobstores_lock:
+            jobs = []
+
+            if pending is not False:
+                for job, alias, replace_existing in self._pending_jobs:
+                    if jobstore is None or alias == jobstore:
+                        jobs.append(job)
+
+            if pending is not True:
+                for alias, store in six.iteritems(self._jobstores):
+                    if jobstore is None or alias == jobstore:
+                        jobs.extend(store.get_all_jobs())
+
+            return jobs
+
+    def get_job(self, job_id, jobstore=None):
+        """
+        Returns the Job that matches the given ``job_id``.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that most likely contains the job
+        :return: the Job by the given ID, or ``None`` if it wasn't found
+        :rtype: Job
+        """
+
+        with self._jobstores_lock:
+            try:
+                return self._lookup_job(job_id, jobstore)[0]
+            except JobLookupError:
+                return
+
+    def remove_job(self, job_id, jobstore=None):
+        """
+        Removes a job, preventing it from being run any more.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        :raises JobLookupError: if the job was not found
+        """
+
+        with self._jobstores_lock:
+            # Check if the job is among the pending jobs
+            for i, (job, jobstore_alias, replace_existing) in enumerate(self._pending_jobs):
+                if job.id == job_id:
+                    del self._pending_jobs[i]
+                    jobstore = jobstore_alias
+                    break
+            else:
+                # Otherwise, try to remove it from each store until it succeeds or we run out of stores to check
+                for alias, store in six.iteritems(self._jobstores):
+                    if jobstore in (None, alias):
+                        try:
+                            store.remove_job(job_id)
+                        except JobLookupError:
+                            continue
+
+                        jobstore = alias
+                        break
+
+        if jobstore is None:
+            raise JobLookupError(job_id)
+
+        # Notify listeners that a job has been removed
+        event = JobEvent(EVENT_JOB_REMOVED, job_id, jobstore)
+        self._dispatch_event(event)
+
+        self._logger.info('Removed job %s', job_id)
+
+    def remove_all_jobs(self, jobstore=None):
+        """
+        Removes all jobs from the specified job store, or all job stores if none is given.
+
+        :param str|unicode jobstore: alias of the job store
+        """
+
+        with self._jobstores_lock:
+            if jobstore:
+                self._pending_jobs = [pending for pending in self._pending_jobs if pending[1] != jobstore]
+            else:
+                self._pending_jobs = []
+
+            for alias, store in six.iteritems(self._jobstores):
+                if jobstore in (None, alias):
+                    store.remove_all_jobs()
+
+        self._dispatch_event(SchedulerEvent(EVENT_ALL_JOBS_REMOVED, jobstore))
+
+    def print_jobs(self, jobstore=None, out=None):
+        """
+        print_jobs(jobstore=None, out=sys.stdout)
+
+        Prints out a textual listing of all jobs currently scheduled on either all job stores or just a specific one.
+
+        :param str|unicode jobstore: alias of the job store, ``None`` to list jobs from all stores
+        :param file out: a file-like object to print to (defaults to **sys.stdout** if nothing is given)
+        """
+
+        out = out or sys.stdout
+        with self._jobstores_lock:
+            if self._pending_jobs:
+                print(six.u('Pending jobs:'), file=out)
+                for job, jobstore_alias, replace_existing in self._pending_jobs:
+                    if jobstore in (None, jobstore_alias):
+                        print(six.u('    %s') % job, file=out)
+
+            for alias, store in six.iteritems(self._jobstores):
+                if jobstore in (None, alias):
+                    print(six.u('Jobstore %s:') % alias, file=out)
+                    jobs = store.get_all_jobs()
+                    if jobs:
+                        for job in jobs:
+                            print(six.u('    %s') % job, file=out)
+                    else:
+                        print(six.u('    No scheduled jobs'), file=out)
+
+    @abstractmethod
+    def wakeup(self):
+        """
+        Notifies the scheduler that there may be jobs due for execution.
+        Triggers :meth:`_process_jobs` to be run in an implementation specific manner.
+        """
+
+    #
+    # Private API
+    #
+
+    def _configure(self, config):
+        # Set general options
+        self._logger = maybe_ref(config.pop('logger', None)) or getLogger('apscheduler.scheduler')
+        self.timezone = astimezone(config.pop('timezone', None)) or get_localzone()
+
+        # Set the job defaults
+        job_defaults = config.get('job_defaults', {})
+        self._job_defaults = {
+            'misfire_grace_time': asint(job_defaults.get('misfire_grace_time', 1)),
+            'coalesce': asbool(job_defaults.get('coalesce', True)),
+            'max_instances': asint(job_defaults.get('max_instances', 1))
+        }
+
+        # Configure executors
+        self._executors.clear()
+        for alias, value in six.iteritems(config.get('executors', {})):
+            if isinstance(value, BaseExecutor):
+                self.add_executor(value, alias)
+            elif isinstance(value, MutableMapping):
+                executor_class = value.pop('class', None)
+                plugin = value.pop('type', None)
+                if plugin:
+                    executor = self._create_plugin_instance('executor', plugin, value)
+                elif executor_class:
+                    cls = maybe_ref(executor_class)
+                    executor = cls(**value)
+                else:
+                    raise ValueError('Cannot create executor "%s" -- either "type" or "class" must be defined' % alias)
+
+                self.add_executor(executor, alias)
+            else:
+                raise TypeError("Expected executor instance or dict for executors['%s'], got %s instead" % (
+                    alias, value.__class__.__name__))
+
+        # Configure job stores
+        self._jobstores.clear()
+        for alias, value in six.iteritems(config.get('jobstores', {})):
+            if isinstance(value, BaseJobStore):
+                self.add_jobstore(value, alias)
+            elif isinstance(value, MutableMapping):
+                jobstore_class = value.pop('class', None)
+                plugin = value.pop('type', None)
+                if plugin:
+                    jobstore = self._create_plugin_instance('jobstore', plugin, value)
+                elif jobstore_class:
+                    cls = maybe_ref(jobstore_class)
+                    jobstore = cls(**value)
+                else:
+                    raise ValueError('Cannot create job store "%s" -- either "type" or "class" must be defined' % alias)
+
+                self.add_jobstore(jobstore, alias)
+            else:
+                raise TypeError("Expected job store instance or dict for jobstores['%s'], got %s instead" % (
+                    alias, value.__class__.__name__))
+
+    def _create_default_executor(self):
+        """Creates a default executor store, specific to the particular scheduler type."""
+
+        return ThreadPoolExecutor()
+
+    def _create_default_jobstore(self):
+        """Creates a default job store, specific to the particular scheduler type."""
+
+        return MemoryJobStore()
+
+    def _lookup_executor(self, alias):
+        """
+        Returns the executor instance by the given name from the list of executors that were added to this scheduler.
+
+        :type alias: str
+        :raises KeyError: if no executor by the given alias is not found
+        """
+
+        try:
+            return self._executors[alias]
+        except KeyError:
+            raise KeyError('No such executor: %s' % alias)
+
+    def _lookup_jobstore(self, alias):
+        """
+        Returns the job store instance by the given name from the list of job stores that were added to this scheduler.
+
+        :type alias: str
+        :raises KeyError: if no job store by the given alias is not found
+        """
+
+        try:
+            return self._jobstores[alias]
+        except KeyError:
+            raise KeyError('No such job store: %s' % alias)
+
+    def _lookup_job(self, job_id, jobstore_alias):
+        """
+        Finds a job by its ID.
+
+        :type job_id: str
+        :param str jobstore_alias: alias of a job store to look in
+        :return tuple[Job, str]: a tuple of job, jobstore alias (jobstore alias is None in case of a pending job)
+        :raises JobLookupError: if no job by the given ID is found.
+        """
+
+        # Check if the job is among the pending jobs
+        for job, alias, replace_existing in self._pending_jobs:
+            if job.id == job_id:
+                return job, None
+
+        # Look in all job stores
+        for alias, store in six.iteritems(self._jobstores):
+            if jobstore_alias in (None, alias):
+                job = store.lookup_job(job_id)
+                if job is not None:
+                    return job, alias
+
+        raise JobLookupError(job_id)
+
+    def _dispatch_event(self, event):
+        """
+        Dispatches the given event to interested listeners.
+
+        :param SchedulerEvent event: the event to send
+        """
+
+        with self._listeners_lock:
+            listeners = tuple(self._listeners)
+
+        for cb, mask in listeners:
+            if event.code & mask:
+                try:
+                    cb(event)
+                except:
+                    self._logger.exception('Error notifying listener')
+
+    def _real_add_job(self, job, jobstore_alias, replace_existing, wakeup):
+        """
+        :param Job job: the job to add
+        :param bool replace_existing: ``True`` to use update_job() in case the job already exists in the store
+        :param bool wakeup: ``True`` to wake up the scheduler after adding the job
+        """
+
+        # Fill in undefined values with defaults
+        replacements = {}
+        for key, value in six.iteritems(self._job_defaults):
+            if not hasattr(job, key):
+                replacements[key] = value
+
+        # Calculate the next run time if there is none defined
+        if not hasattr(job, 'next_run_time'):
+            now = datetime.now(self.timezone)
+            replacements['next_run_time'] = job.trigger.get_next_fire_time(None, now)
+
+        # Apply any replacements
+        job._modify(**replacements)
+
+        # Add the job to the given job store
+        store = self._lookup_jobstore(jobstore_alias)
+        try:
+            store.add_job(job)
+        except ConflictingIdError:
+            if replace_existing:
+                store.update_job(job)
+            else:
+                raise
+
+        # Mark the job as no longer pending
+        job._jobstore_alias = jobstore_alias
+
+        # Notify listeners that a new job has been added
+        event = JobEvent(EVENT_JOB_ADDED, job.id, jobstore_alias)
+        self._dispatch_event(event)
+
+        self._logger.info('Added job "%s" to job store "%s"', job.name, jobstore_alias)
+
+        # Notify the scheduler about the new job
+        if wakeup:
+            self.wakeup()
+
+    def _create_plugin_instance(self, type_, alias, constructor_kwargs):
+        """Creates an instance of the given plugin type, loading the plugin first if necessary."""
+
+        plugin_container, class_container, base_class = {
+            'trigger': (self._trigger_plugins, self._trigger_classes, BaseTrigger),
+            'jobstore': (self._jobstore_plugins, self._jobstore_classes, BaseJobStore),
+            'executor': (self._executor_plugins, self._executor_classes, BaseExecutor)
+        }[type_]
+
+        try:
+            plugin_cls = class_container[alias]
+        except KeyError:
+            if alias in plugin_container:
+                plugin_cls = class_container[alias] = plugin_container[alias].load()
+                if not issubclass(plugin_cls, base_class):
+                    raise TypeError('The {0} entry point does not point to a {0} class'.format(type_))
+            else:
+                raise LookupError('No {0} by the name "{1}" was found'.format(type_, alias))
+
+        return plugin_cls(**constructor_kwargs)
+
+    def _create_trigger(self, trigger, trigger_args):
+        if isinstance(trigger, BaseTrigger):
+            return trigger
+        elif trigger is None:
+            trigger = 'date'
+        elif not isinstance(trigger, six.string_types):
+            raise TypeError('Expected a trigger instance or string, got %s instead' % trigger.__class__.__name__)
+
+        # Use the scheduler's time zone if nothing else is specified
+        trigger_args.setdefault('timezone', self.timezone)
+
+        # Instantiate the trigger class
+        return self._create_plugin_instance('trigger', trigger, trigger_args)
+
+    def _create_lock(self):
+        """Creates a reentrant lock object."""
+
+        return RLock()
+
+    def _process_jobs(self):
+        """
+        Iterates through jobs in every jobstore, starts jobs that are due and figures out how long to wait for the next
+        round.
+        """
+
+        self._logger.debug('Looking for jobs to run')
+        now = datetime.now(self.timezone)
+        next_wakeup_time = None
+
+        with self._jobstores_lock:
+            for jobstore_alias, jobstore in six.iteritems(self._jobstores):
+                for job in jobstore.get_due_jobs(now):
+                    # Look up the job's executor
+                    try:
+                        executor = self._lookup_executor(job.executor)
+                    except:
+                        self._logger.error(
+                            'Executor lookup ("%s") failed for job "%s" -- removing it from the job store',
+                            job.executor, job)
+                        self.remove_job(job.id, jobstore_alias)
+                        continue
+
+                    run_times = job._get_run_times(now)
+                    run_times = run_times[-1:] if run_times and job.coalesce else run_times
+                    if run_times:
+                        try:
+                            executor.submit_job(job, run_times)
+                        except MaxInstancesReachedError:
+                            self._logger.warning(
+                                'Execution of job "%s" skipped: maximum number of running instances reached (%d)',
+                                job, job.max_instances)
+                        except:
+                            self._logger.exception('Error submitting job "%s" to executor "%s"', job, job.executor)
+
+                        # Update the job if it has a next execution time. Otherwise remove it from the job store.
+                        job_next_run = job.trigger.get_next_fire_time(run_times[-1], now)
+                        if job_next_run:
+                            job._modify(next_run_time=job_next_run)
+                            jobstore.update_job(job)
+                        else:
+                            self.remove_job(job.id, jobstore_alias)
+
+                # Set a new next wakeup time if there isn't one yet or the jobstore has an even earlier one
+                jobstore_next_run_time = jobstore.get_next_run_time()
+                if jobstore_next_run_time and (next_wakeup_time is None or jobstore_next_run_time < next_wakeup_time):
+                    next_wakeup_time = jobstore_next_run_time
+
+        # Determine the delay until this method should be called again
+        if next_wakeup_time is not None:
+            wait_seconds = max(timedelta_seconds(next_wakeup_time - now), 0)
+            self._logger.debug('Next wakeup is due at %s (in %f seconds)', next_wakeup_time, wait_seconds)
+        else:
+            wait_seconds = None
+            self._logger.debug('No jobs; waiting until a job is added')
+
+        return wait_seconds

lib/apscheduler/schedulers/blocking.py

@@ -0,0 +1,32 @@
+from __future__ import absolute_import
+from threading import Event
+
+from apscheduler.schedulers.base import BaseScheduler
+
+
+class BlockingScheduler(BaseScheduler):
+    """
+    A scheduler that runs in the foreground (:meth:`~apscheduler.schedulers.base.BaseScheduler.start` will block).
+    """
+
+    MAX_WAIT_TIME = 4294967  # Maximum value accepted by Event.wait() on Windows
+
+    _event = None
+
+    def start(self):
+        super(BlockingScheduler, self).start()
+        self._event = Event()
+        self._main_loop()
+
+    def shutdown(self, wait=True):
+        super(BlockingScheduler, self).shutdown(wait)
+        self._event.set()
+
+    def _main_loop(self):
+        while self.running:
+            wait_seconds = self._process_jobs()
+            self._event.wait(wait_seconds if wait_seconds is not None else self.MAX_WAIT_TIME)
+            self._event.clear()
+
+    def wakeup(self):
+        self._event.set()

lib/apscheduler/schedulers/gevent.py

@@ -0,0 +1,35 @@
+from __future__ import absolute_import
+
+from apscheduler.schedulers.blocking import BlockingScheduler
+from apscheduler.schedulers.base import BaseScheduler
+
+try:
+    from gevent.event import Event
+    from gevent.lock import RLock
+    import gevent
+except ImportError:  # pragma: nocover
+    raise ImportError('GeventScheduler requires gevent installed')
+
+
+class GeventScheduler(BlockingScheduler):
+    """A scheduler that runs as a Gevent greenlet."""
+
+    _greenlet = None
+
+    def start(self):
+        BaseScheduler.start(self)
+        self._event = Event()
+        self._greenlet = gevent.spawn(self._main_loop)
+        return self._greenlet
+
+    def shutdown(self, wait=True):
+        super(GeventScheduler, self).shutdown(wait)
+        self._greenlet.join()
+        del self._greenlet
+
+    def _create_lock(self):
+        return RLock()
+
+    def _create_default_executor(self):
+        from apscheduler.executors.gevent import GeventExecutor
+        return GeventExecutor()

lib/apscheduler/schedulers/qt.py

@@ -0,0 +1,46 @@
+from __future__ import absolute_import
+
+from apscheduler.schedulers.base import BaseScheduler
+
+try:
+    from PyQt5.QtCore import QObject, QTimer
+except ImportError:  # pragma: nocover
+    try:
+        from PyQt4.QtCore import QObject, QTimer
+    except ImportError:
+        try:
+            from PySide.QtCore import QObject, QTimer  # flake8: noqa
+        except ImportError:
+            raise ImportError('QtScheduler requires either PyQt5, PyQt4 or PySide installed')
+
+
+class QtScheduler(BaseScheduler):
+    """A scheduler that runs in a Qt event loop."""
+
+    _timer = None
+
+    def start(self):
+        super(QtScheduler, self).start()
+        self.wakeup()
+
+    def shutdown(self, wait=True):
+        super(QtScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._timer = QTimer.singleShot(wait_seconds * 1000, self._process_jobs)
+
+    def _stop_timer(self):
+        if self._timer:
+            if self._timer.isActive():
+                self._timer.stop()
+            del self._timer
+
+    def wakeup(self):
+        self._start_timer(0)
+
+    def _process_jobs(self):
+        wait_seconds = super(QtScheduler, self)._process_jobs()
+        self._start_timer(wait_seconds)

lib/apscheduler/schedulers/tornado.py

@@ -0,0 +1,60 @@
+from __future__ import absolute_import
+from datetime import timedelta
+from functools import wraps
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.util import maybe_ref
+
+try:
+    from tornado.ioloop import IOLoop
+except ImportError:  # pragma: nocover
+    raise ImportError('TornadoScheduler requires tornado installed')
+
+
+def run_in_ioloop(func):
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        self._ioloop.add_callback(func, self, *args, **kwargs)
+    return wrapper
+
+
+class TornadoScheduler(BaseScheduler):
+    """
+    A scheduler that runs on a Tornado IOLoop.
+
+    =========== ===============================================================
+    ``io_loop`` Tornado IOLoop instance to use (defaults to the global IO loop)
+    =========== ===============================================================
+    """
+
+    _ioloop = None
+    _timeout = None
+
+    def start(self):
+        super(TornadoScheduler, self).start()
+        self.wakeup()
+
+    @run_in_ioloop
+    def shutdown(self, wait=True):
+        super(TornadoScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _configure(self, config):
+        self._ioloop = maybe_ref(config.pop('io_loop', None)) or IOLoop.current()
+        super(TornadoScheduler, self)._configure(config)
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._timeout = self._ioloop.add_timeout(timedelta(seconds=wait_seconds), self.wakeup)
+
+    def _stop_timer(self):
+        if self._timeout:
+            self._ioloop.remove_timeout(self._timeout)
+            del self._timeout
+
+    @run_in_ioloop
+    def wakeup(self):
+        self._stop_timer()
+        wait_seconds = self._process_jobs()
+        self._start_timer(wait_seconds)

lib/apscheduler/schedulers/twisted.py

@@ -0,0 +1,65 @@
+from __future__ import absolute_import
+from functools import wraps
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.util import maybe_ref
+
+try:
+    from twisted.internet import reactor as default_reactor
+except ImportError:  # pragma: nocover
+    raise ImportError('TwistedScheduler requires Twisted installed')
+
+
+def run_in_reactor(func):
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        self._reactor.callFromThread(func, self, *args, **kwargs)
+    return wrapper
+
+
+class TwistedScheduler(BaseScheduler):
+    """
+    A scheduler that runs on a Twisted reactor.
+
+    Extra options:
+
+    =========== ========================================================
+    ``reactor`` Reactor instance to use (defaults to the global reactor)
+    =========== ========================================================
+    """
+
+    _reactor = None
+    _delayedcall = None
+
+    def _configure(self, config):
+        self._reactor = maybe_ref(config.pop('reactor', default_reactor))
+        super(TwistedScheduler, self)._configure(config)
+
+    def start(self):
+        super(TwistedScheduler, self).start()
+        self.wakeup()
+
+    @run_in_reactor
+    def shutdown(self, wait=True):
+        super(TwistedScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._delayedcall = self._reactor.callLater(wait_seconds, self.wakeup)
+
+    def _stop_timer(self):
+        if self._delayedcall and self._delayedcall.active():
+            self._delayedcall.cancel()
+            del self._delayedcall
+
+    @run_in_reactor
+    def wakeup(self):
+        self._stop_timer()
+        wait_seconds = self._process_jobs()
+        self._start_timer(wait_seconds)
+
+    def _create_default_executor(self):
+        from apscheduler.executors.twisted import TwistedExecutor
+        return TwistedExecutor()

lib/apscheduler/threadpool.py

@@ -1,133 +0,0 @@
-"""
-Generic thread pool class. Modeled after Java's ThreadPoolExecutor.
-Please note that this ThreadPool does *not* fully implement the PEP 3148
-ThreadPool!
-"""
-
-from threading import Thread, Lock, currentThread
-from weakref import ref
-import logging
-import atexit
-
-try:
-    from queue import Queue, Empty
-except ImportError:
-    from Queue import Queue, Empty
-
-logger = logging.getLogger(__name__)
-_threadpools = set()
-
-
-# Worker threads are daemonic in order to let the interpreter exit without
-# an explicit shutdown of the thread pool. The following trick is necessary
-# to allow worker threads to finish cleanly.
-def _shutdown_all():
-    for pool_ref in tuple(_threadpools):
-        pool = pool_ref()
-        if pool:
-            pool.shutdown()
-
-atexit.register(_shutdown_all)
-
-
-class ThreadPool(object):
-    def __init__(self, core_threads=0, max_threads=20, keepalive=1):
-        """
-        :param core_threads: maximum number of persistent threads in the pool
-        :param max_threads: maximum number of total threads in the pool
-        :param thread_class: callable that creates a Thread object
-        :param keepalive: seconds to keep non-core worker threads waiting
-            for new tasks
-        """
-        self.core_threads = core_threads
-        self.max_threads = max(max_threads, core_threads, 1)
-        self.keepalive = keepalive
-        self._queue = Queue()
-        self._threads_lock = Lock()
-        self._threads = set()
-        self._shutdown = False
-
-        _threadpools.add(ref(self))
-        logger.info('Started thread pool with %d core threads and %s maximum '
-                    'threads', core_threads, max_threads or 'unlimited')
-
-    def _adjust_threadcount(self):
-        self._threads_lock.acquire()
-        try:
-            if self.num_threads < self.max_threads:
-                self._add_thread(self.num_threads < self.core_threads)
-        finally:
-            self._threads_lock.release()
-
-    def _add_thread(self, core):
-        t = Thread(target=self._run_jobs, args=(core,))
-        t.setDaemon(True)
-        t.start()
-        self._threads.add(t)
-
-    def _run_jobs(self, core):
-        logger.debug('Started worker thread')
-        block = True
-        timeout = None
-        if not core:
-            block = self.keepalive > 0
-            timeout = self.keepalive
-
-        while True:
-            try:
-                func, args, kwargs = self._queue.get(block, timeout)
-            except Empty:
-                break
-
-            if self._shutdown:
-                break
-
-            try:
-                func(*args, **kwargs)
-            except:
-                logger.exception('Error in worker thread')
-
-        self._threads_lock.acquire()
-        self._threads.remove(currentThread())
-        self._threads_lock.release()
-
-        logger.debug('Exiting worker thread')
-
-    @property
-    def num_threads(self):
-        return len(self._threads)
-
-    def submit(self, func, *args, **kwargs):
-        if self._shutdown:
-            raise RuntimeError('Cannot schedule new tasks after shutdown')
-
-        self._queue.put((func, args, kwargs))
-        self._adjust_threadcount()
-
-    def shutdown(self, wait=True):
-        if self._shutdown:
-            return
-
-        logging.info('Shutting down thread pool')
-        self._shutdown = True
-        _threadpools.remove(ref(self))
-
-        self._threads_lock.acquire()
-        for _ in range(self.num_threads):
-            self._queue.put((None, None, None))
-        self._threads_lock.release()
-
-        if wait:
-            self._threads_lock.acquire()
-            threads = tuple(self._threads)
-            self._threads_lock.release()
-            for thread in threads:
-                thread.join()
-
-    def __repr__(self):
-        if self.max_threads:
-            threadcount = '%d/%d' % (self.num_threads, self.max_threads)
-        else:
-            threadcount = '%d' % self.num_threads
-
-        return '<ThreadPool at %x; threads=%s>' % (id(self), threadcount)

lib/apscheduler/triggers/__init__.py

@@ -1,3 +0,0 @@
-from apscheduler.triggers.cron import CronTrigger
-from apscheduler.triggers.interval import IntervalTrigger
-from apscheduler.triggers.simple import SimpleTrigger

lib/apscheduler/triggers/base.py

@@ -0,0 +1,16 @@
+from abc import ABCMeta, abstractmethod
+
+import six
+
+
+class BaseTrigger(six.with_metaclass(ABCMeta)):
+    """Abstract base class that defines the interface that every trigger must implement."""
+
+    @abstractmethod
+    def get_next_fire_time(self, previous_fire_time, now):
+        """
+        Returns the next datetime to fire on, If no such datetime can be calculated, returns ``None``.
+
+        :param datetime.datetime previous_fire_time: the previous time the trigger was fired
+        :param datetime.datetime now: current datetime
+        """

lib/apscheduler/triggers/cron/__init__.py

@@ -1,32 +1,71 @@
-from datetime import date, datetime
+from datetime import datetime, timedelta
 
-from apscheduler.triggers.cron.fields import *
+from tzlocal import get_localzone
-from apscheduler.util import datetime_ceil, convert_to_datetime
+import six
+
+from apscheduler.triggers.base import BaseTrigger
+from apscheduler.triggers.cron.fields import BaseField, WeekField, DayOfMonthField, DayOfWeekField, DEFAULT_VALUES
+from apscheduler.util import datetime_ceil, convert_to_datetime, datetime_repr, astimezone
 
 
-class CronTrigger(object):
+class CronTrigger(BaseTrigger):
-    FIELD_NAMES = ('year', 'month', 'day', 'week', 'day_of_week', 'hour',
+    """
-                   'minute', 'second')
+    Triggers when current time matches all specified time constraints, similarly to how the UNIX cron scheduler works.
-    FIELDS_MAP = {'year': BaseField,
+
+    :param int|str year: 4-digit year
+    :param int|str month: month (1-12)
+    :param int|str day: day of the (1-31)
+    :param int|str week: ISO week (1-53)
+    :param int|str day_of_week: number or name of weekday (0-6 or mon,tue,wed,thu,fri,sat,sun)
+    :param int|str hour: hour (0-23)
+    :param int|str minute: minute (0-59)
+    :param int|str second: second (0-59)
+    :param datetime|str start_date: earliest possible date/time to trigger on (inclusive)
+    :param datetime|str end_date: latest possible date/time to trigger on (inclusive)
+    :param datetime.tzinfo|str timezone: time zone to use for the date/time calculations
+                                         (defaults to scheduler timezone)
+
+    .. note:: The first weekday is always **monday**.
+    """
+
+    FIELD_NAMES = ('year', 'month', 'day', 'week', 'day_of_week', 'hour', 'minute', 'second')
+    FIELDS_MAP = {
+        'year': BaseField,
         'month': BaseField,
         'week': WeekField,
         'day': DayOfMonthField,
         'day_of_week': DayOfWeekField,
         'hour': BaseField,
         'minute': BaseField,
-                  'second': BaseField}
+        'second': BaseField
+    }
 
-    def __init__(self, **values):
+    __slots__ = 'timezone', 'start_date', 'end_date', 'fields'
-        self.start_date = values.pop('start_date', None)
-        if self.start_date:
-            self.start_date = convert_to_datetime(self.start_date)
 
+    def __init__(self, year=None, month=None, day=None, week=None, day_of_week=None, hour=None, minute=None,
+                 second=None, start_date=None, end_date=None, timezone=None):
+        if timezone:
+            self.timezone = astimezone(timezone)
+        elif start_date and start_date.tzinfo:
+            self.timezone = start_date.tzinfo
+        elif end_date and end_date.tzinfo:
+            self.timezone = end_date.tzinfo
+        else:
+            self.timezone = get_localzone()
+
+        self.start_date = convert_to_datetime(start_date, self.timezone, 'start_date')
+        self.end_date = convert_to_datetime(end_date, self.timezone, 'end_date')
+
+        values = dict((key, value) for (key, value) in six.iteritems(locals())
+                      if key in self.FIELD_NAMES and value is not None)
         self.fields = []
+        assign_defaults = False
         for field_name in self.FIELD_NAMES:
             if field_name in values:
                 exprs = values.pop(field_name)
                 is_default = False
-            elif not values:
+                assign_defaults = not values
+            elif assign_defaults:
                 exprs = DEFAULT_VALUES[field_name]
                 is_default = True
             else:
@@ -39,18 +78,16 @@ class CronTrigger(object):
 
     def _increment_field_value(self, dateval, fieldnum):
         """
-        Increments the designated field and resets all less significant fields
+        Increments the designated field and resets all less significant fields to their minimum values.
-        to their minimum values.
 
         :type dateval: datetime
         :type fieldnum: int
-        :type amount: int
+        :return: a tuple containing the new date, and the number of the field that was actually incremented
         :rtype: tuple
-        :return: a tuple containing the new date, and the number of the field
-                 that was actually incremented
         """
-        i = 0
+
         values = {}
+        i = 0
         while i < len(self.fields):
             field = self.fields[i]
             if not field.REAL:
@@ -77,7 +114,8 @@ class CronTrigger(object):
                     values[field.name] = value + 1
                     i += 1
 
-        return datetime(**values), fieldnum
+        difference = datetime(**values) - dateval.replace(tzinfo=None)
+        return self.timezone.normalize(dateval + difference), fieldnum
 
     def _set_field_value(self, dateval, fieldnum, new_value):
         values = {}
@@ -90,13 +128,17 @@ class CronTrigger(object):
                 else:
                     values[field.name] = new_value
 
-        return datetime(**values)
+        difference = datetime(**values) - dateval.replace(tzinfo=None)
+        return self.timezone.normalize(dateval + difference)
+
+    def get_next_fire_time(self, previous_fire_time, now):
+        if previous_fire_time:
+            start_date = max(now, previous_fire_time + timedelta(microseconds=1))
+        else:
+            start_date = max(now, self.start_date) if self.start_date else now
 
-    def get_next_fire_time(self, start_date):
-        if self.start_date:
-            start_date = max(start_date, self.start_date)
-        next_date = datetime_ceil(start_date)
         fieldnum = 0
+        next_date = datetime_ceil(start_date).astimezone(self.timezone)
         while 0 <= fieldnum < len(self.fields):
             field = self.fields[fieldnum]
             curr_value = field.get_value(next_date)
@@ -104,32 +146,31 @@ class CronTrigger(object):
 
             if next_value is None:
                 # No valid value was found
-                next_date, fieldnum = self._increment_field_value(next_date,
+                next_date, fieldnum = self._increment_field_value(next_date, fieldnum - 1)
-                                                                  fieldnum - 1)
             elif next_value > curr_value:
                 # A valid, but higher than the starting value, was found
                 if field.REAL:
-                    next_date = self._set_field_value(next_date, fieldnum,
+                    next_date = self._set_field_value(next_date, fieldnum, next_value)
-                                                      next_value)
                     fieldnum += 1
                 else:
-                    next_date, fieldnum = self._increment_field_value(next_date,
+                    next_date, fieldnum = self._increment_field_value(next_date, fieldnum)
-                                                                      fieldnum)
             else:
                 # A valid value was found, no changes necessary
                 fieldnum += 1
 
+            # Return if the date has rolled past the end date
+            if self.end_date and next_date > self.end_date:
+                return None
+
         if fieldnum >= 0:
             return next_date
 
     def __str__(self):
-        options = ["%s='%s'" % (f.name, str(f)) for f in self.fields
+        options = ["%s='%s'" % (f.name, f) for f in self.fields if not f.is_default]
-                   if not f.is_default]
         return 'cron[%s]' % (', '.join(options))
 
     def __repr__(self):
-        options = ["%s='%s'" % (f.name, str(f)) for f in self.fields
+        options = ["%s='%s'" % (f.name, f) for f in self.fields if not f.is_default]
-                   if not f.is_default]
         if self.start_date:
-            options.append("start_date='%s'" % self.start_date.isoformat(' '))
+            options.append("start_date='%s'" % datetime_repr(self.start_date))
         return '<%s (%s)>' % (self.__class__.__name__, ', '.join(options))

lib/apscheduler/triggers/cron/expressions.py

@@ -7,8 +7,8 @@ import re
 
 from apscheduler.util import asint
 
-__all__ = ('AllExpression', 'RangeExpression', 'WeekdayRangeExpression',
+__all__ = ('AllExpression', 'RangeExpression', 'WeekdayRangeExpression', 'WeekdayPositionExpression',
-           'WeekdayPositionExpression')
+           'LastDayOfMonthExpression')
 
 
 WEEKDAYS = ['mon', 'tue', 'wed', 'thu', 'fri', 'sat', 'sun']
@@ -57,8 +57,7 @@ class RangeExpression(AllExpression):
         if last is None and step is None:
             last = first
         if last is not None and first > last:
-            raise ValueError('The minimum value in a range must not be '
+            raise ValueError('The minimum value in a range must not be higher than the maximum')
-                             'higher than the maximum')
         self.first = first
         self.last = last
 
@@ -102,8 +101,7 @@ class RangeExpression(AllExpression):
 
 
 class WeekdayRangeExpression(RangeExpression):
-    value_re = re.compile(r'(?P<first>[a-z]+)(?:-(?P<last>[a-z]+))?',
+    value_re = re.compile(r'(?P<first>[a-z]+)(?:-(?P<last>[a-z]+))?', re.IGNORECASE)
-                          re.IGNORECASE)
 
     def __init__(self, first, last=None):
         try:
@@ -135,8 +133,7 @@ class WeekdayRangeExpression(RangeExpression):
 
 class WeekdayPositionExpression(AllExpression):
     options = ['1st', '2nd', '3rd', '4th', '5th', 'last']
-    value_re = re.compile(r'(?P<option_name>%s) +(?P<weekday_name>(?:\d+|\w+))'
+    value_re = re.compile(r'(?P<option_name>%s) +(?P<weekday_name>(?:\d+|\w+))' % '|'.join(options), re.IGNORECASE)
-                          % '|'.join(options), re.IGNORECASE)
 
     def __init__(self, option_name, weekday_name):
         try:
@@ -169,10 +166,23 @@ class WeekdayPositionExpression(AllExpression):
             return target_day
 
     def __str__(self):
-        return '%s %s' % (self.options[self.option_num],
+        return '%s %s' % (self.options[self.option_num], WEEKDAYS[self.weekday])
-                          WEEKDAYS[self.weekday])
 
     def __repr__(self):
-        return "%s('%s', '%s')" % (self.__class__.__name__,
+        return "%s('%s', '%s')" % (self.__class__.__name__, self.options[self.option_num], WEEKDAYS[self.weekday])
-                                   self.options[self.option_num],
+
-                                   WEEKDAYS[self.weekday])
+
+class LastDayOfMonthExpression(AllExpression):
+    value_re = re.compile(r'last', re.IGNORECASE)
+
+    def __init__(self):
+        pass
+
+    def get_next_value(self, date, field):
+        return monthrange(date.year, date.month)[1]
+
+    def __str__(self):
+        return 'last'
+
+    def __repr__(self):
+        return "%s()" % self.__class__.__name__

lib/apscheduler/triggers/cron/fields.py

@@ -5,18 +5,18 @@ fields.
 
 from calendar import monthrange
 
-from apscheduler.triggers.cron.expressions import *
+from apscheduler.triggers.cron.expressions import (
-
+    AllExpression, RangeExpression, WeekdayPositionExpression, LastDayOfMonthExpression, WeekdayRangeExpression)
-__all__ = ('MIN_VALUES', 'MAX_VALUES', 'DEFAULT_VALUES', 'BaseField',
-           'WeekField', 'DayOfMonthField', 'DayOfWeekField')
 
 
-MIN_VALUES = {'year': 1970, 'month': 1, 'day': 1, 'week': 1,
+__all__ = ('MIN_VALUES', 'MAX_VALUES', 'DEFAULT_VALUES', 'BaseField', 'WeekField', 'DayOfMonthField', 'DayOfWeekField')
-              'day_of_week': 0, 'hour': 0, 'minute': 0, 'second': 0}
+
-MAX_VALUES = {'year': 2 ** 63, 'month': 12, 'day:': 31, 'week': 53,
+
-              'day_of_week': 6, 'hour': 23, 'minute': 59, 'second': 59}
+MIN_VALUES = {'year': 1970, 'month': 1, 'day': 1, 'week': 1, 'day_of_week': 0, 'hour': 0, 'minute': 0, 'second': 0}
-DEFAULT_VALUES = {'year': '*', 'month': 1, 'day': 1, 'week': '*',
+MAX_VALUES = {'year': 2 ** 63, 'month': 12, 'day:': 31, 'week': 53, 'day_of_week': 6, 'hour': 23, 'minute': 59,
-                  'day_of_week': '*', 'hour': 0, 'minute': 0, 'second': 0}
+              'second': 59}
+DEFAULT_VALUES = {'year': '*', 'month': 1, 'day': 1, 'week': '*', 'day_of_week': '*', 'hour': 0, 'minute': 0,
+                  'second': 0}
 
 
 class BaseField(object):
@@ -65,16 +65,14 @@ class BaseField(object):
                 self.expressions.append(compiled_expr)
                 return
 
-        raise ValueError('Unrecognized expression "%s" for field "%s"' %
+        raise ValueError('Unrecognized expression "%s" for field "%s"' % (expr, self.name))
-                         (expr, self.name))
 
     def __str__(self):
         expr_strings = (str(e) for e in self.expressions)
         return ','.join(expr_strings)
 
     def __repr__(self):
-        return "%s('%s', '%s')" % (self.__class__.__name__, self.name,
+        return "%s('%s', '%s')" % (self.__class__.__name__, self.name, self)
-                                   str(self))
 
 
 class WeekField(BaseField):
@@ -85,7 +83,7 @@ class WeekField(BaseField):
 
 
 class DayOfMonthField(BaseField):
-    COMPILERS = BaseField.COMPILERS + [WeekdayPositionExpression]
+    COMPILERS = BaseField.COMPILERS + [WeekdayPositionExpression, LastDayOfMonthExpression]
 
     def get_max(self, dateval):
         return monthrange(dateval.year, dateval.month)[1]

lib/apscheduler/triggers/date.py

@@ -0,0 +1,30 @@
+from datetime import datetime
+
+from tzlocal import get_localzone
+
+from apscheduler.triggers.base import BaseTrigger
+from apscheduler.util import convert_to_datetime, datetime_repr, astimezone
+
+
+class DateTrigger(BaseTrigger):
+    """
+    Triggers once on the given datetime. If ``run_date`` is left empty, current time is used.
+
+    :param datetime|str run_date: the date/time to run the job at
+    :param datetime.tzinfo|str timezone: time zone for ``run_date`` if it doesn't have one already
+    """
+
+    __slots__ = 'timezone', 'run_date'
+
+    def __init__(self, run_date=None, timezone=None):
+        timezone = astimezone(timezone) or get_localzone()
+        self.run_date = convert_to_datetime(run_date or datetime.now(), timezone, 'run_date')
+
+    def get_next_fire_time(self, previous_fire_time, now):
+        return self.run_date if previous_fire_time is None else None
+
+    def __str__(self):
+        return 'date[%s]' % datetime_repr(self.run_date)
+
+    def __repr__(self):
+        return "<%s (run_date='%s')>" % (self.__class__.__name__, datetime_repr(self.run_date))

lib/apscheduler/triggers/interval.py

@@ -1,39 +1,65 @@
-from datetime import datetime, timedelta
+from datetime import timedelta, datetime
 from math import ceil
 
-from apscheduler.util import convert_to_datetime, timedelta_seconds
+from tzlocal import get_localzone
+
+from apscheduler.triggers.base import BaseTrigger
+from apscheduler.util import convert_to_datetime, timedelta_seconds, datetime_repr, astimezone
 
 
-class IntervalTrigger(object):
+class IntervalTrigger(BaseTrigger):
-    def __init__(self, interval, start_date=None):
+    """
-        if not isinstance(interval, timedelta):
+    Triggers on specified intervals, starting on ``start_date`` if specified, ``datetime.now()`` + interval
-            raise TypeError('interval must be a timedelta')
+    otherwise.
-        if start_date:
-            start_date = convert_to_datetime(start_date)
 
-        self.interval = interval
+    :param int weeks: number of weeks to wait
+    :param int days: number of days to wait
+    :param int hours: number of hours to wait
+    :param int minutes: number of minutes to wait
+    :param int seconds: number of seconds to wait
+    :param datetime|str start_date: starting point for the interval calculation
+    :param datetime|str end_date: latest possible date/time to trigger on
+    :param datetime.tzinfo|str timezone: time zone to use for the date/time calculations
+    """
+
+    __slots__ = 'timezone', 'start_date', 'end_date', 'interval'
+
+    def __init__(self, weeks=0, days=0, hours=0, minutes=0, seconds=0, start_date=None, end_date=None, timezone=None):
+        self.interval = timedelta(weeks=weeks, days=days, hours=hours, minutes=minutes, seconds=seconds)
         self.interval_length = timedelta_seconds(self.interval)
         if self.interval_length == 0:
             self.interval = timedelta(seconds=1)
             self.interval_length = 1
 
-        if start_date is None:
+        if timezone:
-            self.start_date = datetime.now() + self.interval
+            self.timezone = astimezone(timezone)
+        elif start_date and start_date.tzinfo:
+            self.timezone = start_date.tzinfo
+        elif end_date and end_date.tzinfo:
+            self.timezone = end_date.tzinfo
         else:
-            self.start_date = convert_to_datetime(start_date)
+            self.timezone = get_localzone()
 
-    def get_next_fire_time(self, start_date):
+        start_date = start_date or (datetime.now(self.timezone) + self.interval)
-        if start_date < self.start_date:
+        self.start_date = convert_to_datetime(start_date, self.timezone, 'start_date')
-            return self.start_date
+        self.end_date = convert_to_datetime(end_date, self.timezone, 'end_date')
 
-        timediff_seconds = timedelta_seconds(start_date - self.start_date)
+    def get_next_fire_time(self, previous_fire_time, now):
+        if previous_fire_time:
+            next_fire_time = previous_fire_time + self.interval
+        elif self.start_date > now:
+            next_fire_time = self.start_date
+        else:
+            timediff_seconds = timedelta_seconds(now - self.start_date)
             next_interval_num = int(ceil(timediff_seconds / self.interval_length))
-        return self.start_date + self.interval * next_interval_num
+            next_fire_time = self.start_date + self.interval * next_interval_num
+
+        if not self.end_date or next_fire_time <= self.end_date:
+            return self.timezone.normalize(next_fire_time)
 
     def __str__(self):
         return 'interval[%s]' % str(self.interval)
 
     def __repr__(self):
-        return "<%s (interval=%s, start_date=%s)>" % (
+        return "<%s (interval=%r, start_date='%s')>" % (self.__class__.__name__, self.interval,
-            self.__class__.__name__, repr(self.interval),
+                                                        datetime_repr(self.start_date))
-            repr(self.start_date))

lib/apscheduler/triggers/simple.py

@@ -1,17 +0,0 @@
-from apscheduler.util import convert_to_datetime
-
-
-class SimpleTrigger(object):
-    def __init__(self, run_date):
-        self.run_date = convert_to_datetime(run_date)
-
-    def get_next_fire_time(self, start_date):
-        if self.run_date >= start_date:
-            return self.run_date
-
-    def __str__(self):
-        return 'date[%s]' % str(self.run_date)
-
-    def __repr__(self):
-        return '<%s (run_date=%s)>' % (
-            self.__class__.__name__, repr(self.run_date))

lib/apscheduler/util.py

@@ -1,26 +1,48 @@
-"""
+"""This module contains several handy functions primarily meant for internal use."""
-This module contains several handy functions primarily meant for internal use.
-"""
 
-from datetime import date, datetime, timedelta
+from __future__ import division
-from time import mktime
+from datetime import date, datetime, time, timedelta, tzinfo
+from inspect import isfunction, ismethod, getargspec
+from calendar import timegm
 import re
-import sys
 
-__all__ = ('asint', 'asbool', 'convert_to_datetime', 'timedelta_seconds',
+from pytz import timezone, utc
-           'time_difference', 'datetime_ceil', 'combine_opts',
+import six
-           'get_callable_name', 'obj_to_ref', 'ref_to_obj', 'maybe_ref',
+
-           'to_unicode', 'iteritems', 'itervalues', 'xrange')
+try:
+    from inspect import signature
+except ImportError:  # pragma: nocover
+    try:
+        from funcsigs import signature
+    except ImportError:
+        signature = None
+
+__all__ = ('asint', 'asbool', 'astimezone', 'convert_to_datetime', 'datetime_to_utc_timestamp',
+           'utc_timestamp_to_datetime', 'timedelta_seconds', 'datetime_ceil', 'get_callable_name', 'obj_to_ref',
+           'ref_to_obj', 'maybe_ref', 'repr_escape', 'check_callable_args')
+
+
+class _Undefined(object):
+    def __nonzero__(self):
+        return False
+
+    def __bool__(self):
+        return False
+
+    def __repr__(self):
+        return '<undefined>'
+
+undefined = _Undefined()  #: a unique object that only signifies that no value is defined
 
 
 def asint(text):
     """
-    Safely converts a string to an integer, returning None if the string
+    Safely converts a string to an integer, returning None if the string is None.
-    is None.
 
     :type text: str
     :rtype: int
     """
+
     if text is not None:
         return int(text)
 
@@ -31,6 +53,7 @@ def asbool(obj):
 
     :rtype: bool
     """
+
     if isinstance(obj, str):
         obj = obj.strip().lower()
         if obj in ('true', 'yes', 'on', 'y', 't', '1'):
@@ -41,36 +64,99 @@ def asbool(obj):
     return bool(obj)
 
 
+def astimezone(obj):
+    """
+    Interprets an object as a timezone.
+
+    :rtype: tzinfo
+    """
+
+    if isinstance(obj, six.string_types):
+        return timezone(obj)
+    if isinstance(obj, tzinfo):
+        if not hasattr(obj, 'localize') or not hasattr(obj, 'normalize'):
+            raise TypeError('Only timezones from the pytz library are supported')
+        if obj.zone == 'local':
+            raise ValueError('Unable to determine the name of the local timezone -- use an explicit timezone instead')
+        return obj
+    if obj is not None:
+        raise TypeError('Expected tzinfo, got %s instead' % obj.__class__.__name__)
+
+
 _DATE_REGEX = re.compile(
     r'(?P<year>\d{4})-(?P<month>\d{1,2})-(?P<day>\d{1,2})'
     r'(?: (?P<hour>\d{1,2}):(?P<minute>\d{1,2}):(?P<second>\d{1,2})'
     r'(?:\.(?P<microsecond>\d{1,6}))?)?')
 
 
-def convert_to_datetime(input):
+def convert_to_datetime(input, tz, arg_name):
     """
-    Converts the given object to a datetime object, if possible.
+    Converts the given object to a timezone aware datetime object.
-    If an actual datetime object is passed, it is returned unmodified.
+    If a timezone aware datetime object is passed, it is returned unmodified.
-    If the input is a string, it is parsed as a datetime.
+    If a native datetime object is passed, it is given the specified timezone.
+    If the input is a string, it is parsed as a datetime with the given timezone.
 
     Date strings are accepted in three different forms: date only (Y-m-d),
     date with time (Y-m-d H:M:S) or with date+time with microseconds
     (Y-m-d H:M:S.micro).
 
+    :param str|datetime input: the datetime or string to convert to a timezone aware datetime
+    :param datetime.tzinfo tz: timezone to interpret ``input`` in
+    :param str arg_name: the name of the argument (used in an error message)
     :rtype: datetime
     """
-    if isinstance(input, datetime):
+
-        return input
+    if input is None:
+        return
+    elif isinstance(input, datetime):
+        datetime_ = input
     elif isinstance(input, date):
-        return datetime.fromordinal(input.toordinal())
+        datetime_ = datetime.combine(input, time())
-    elif isinstance(input, str):
+    elif isinstance(input, six.string_types):
         m = _DATE_REGEX.match(input)
         if not m:
             raise ValueError('Invalid date string')
         values = [(k, int(v or 0)) for k, v in m.groupdict().items()]
         values = dict(values)
-        return datetime(**values)
+        datetime_ = datetime(**values)
-    raise TypeError('Unsupported input type: %s' % type(input))
+    else:
+        raise TypeError('Unsupported type for %s: %s' % (arg_name, input.__class__.__name__))
+
+    if datetime_.tzinfo is not None:
+        return datetime_
+    if tz is None:
+        raise ValueError('The "tz" argument must be specified if %s has no timezone information' % arg_name)
+    if isinstance(tz, six.string_types):
+        tz = timezone(tz)
+
+    try:
+        return tz.localize(datetime_, is_dst=None)
+    except AttributeError:
+        raise TypeError('Only pytz timezones are supported (need the localize() and normalize() methods)')
+
+
+def datetime_to_utc_timestamp(timeval):
+    """
+    Converts a datetime instance to a timestamp.
+
+    :type timeval: datetime
+    :rtype: float
+    """
+
+    if timeval is not None:
+        return timegm(timeval.utctimetuple()) + timeval.microsecond / 1000000
+
+
+def utc_timestamp_to_datetime(timestamp):
+    """
+    Converts the given timestamp to a datetime instance.
+
+    :type timestamp: float
+    :rtype: datetime
+    """
+
+    if timestamp is not None:
+        return datetime.fromtimestamp(timestamp, utc)
 
 
 def timedelta_seconds(delta):
@@ -80,125 +166,220 @@ def timedelta_seconds(delta):
     :type delta: timedelta
     :rtype: float
     """
+
     return delta.days * 24 * 60 * 60 + delta.seconds + \
         delta.microseconds / 1000000.0
 
 
-def time_difference(date1, date2):
-    """
-    Returns the time difference in seconds between the given two
-    datetime objects. The difference is calculated as: date1 - date2.
-
-    :param date1: the later datetime
-    :type date1: datetime
-    :param date2: the earlier datetime
-    :type date2: datetime
-    :rtype: float
-    """
-    later = mktime(date1.timetuple()) + date1.microsecond / 1000000.0
-    earlier = mktime(date2.timetuple()) + date2.microsecond / 1000000.0
-    return later - earlier
-
-
 def datetime_ceil(dateval):
     """
     Rounds the given datetime object upwards.
 
     :type dateval: datetime
     """
+
     if dateval.microsecond > 0:
-        return dateval + timedelta(seconds=1,
+        return dateval + timedelta(seconds=1, microseconds=-dateval.microsecond)
-                                   microseconds=-dateval.microsecond)
     return dateval
 
 
-def combine_opts(global_config, prefix, local_config={}):
+def datetime_repr(dateval):
-    """
+    return dateval.strftime('%Y-%m-%d %H:%M:%S %Z') if dateval else 'None'
-    Returns a subdictionary from keys and values of  ``global_config`` where
-    the key starts with the given prefix, combined with options from
-    local_config. The keys in the subdictionary have the prefix removed.
-
-    :type global_config: dict
-    :type prefix: str
-    :type local_config: dict
-    :rtype: dict
-    """
-    prefixlen = len(prefix)
-    subconf = {}
-    for key, value in global_config.items():
-        if key.startswith(prefix):
-            key = key[prefixlen:]
-            subconf[key] = value
-    subconf.update(local_config)
-    return subconf
 
 
 def get_callable_name(func):
     """
     Returns the best available display name for the given function/callable.
+
+    :rtype: str
     """
-    name = func.__module__
+
-    if hasattr(func, '__self__') and func.__self__:
+    # the easy case (on Python 3.3+)
-        name += '.' + func.__self__.__name__
+    if hasattr(func, '__qualname__'):
-    elif hasattr(func, 'im_self') and func.im_self:     # py2.4, 2.5
+        return func.__qualname__
-        name += '.' + func.im_self.__name__
+
+    # class methods, bound and unbound methods
+    f_self = getattr(func, '__self__', None) or getattr(func, 'im_self', None)
+    if f_self and hasattr(func, '__name__'):
+        f_class = f_self if isinstance(f_self, type) else f_self.__class__
+    else:
+        f_class = getattr(func, 'im_class', None)
+
+    if f_class and hasattr(func, '__name__'):
+        return '%s.%s' % (f_class.__name__, func.__name__)
+
+    # class or class instance
+    if hasattr(func, '__call__'):
+        # class
         if hasattr(func, '__name__'):
-        name += '.' + func.__name__
+            return func.__name__
-    return name
+
+        # instance of a class with a __call__ method
+        return func.__class__.__name__
+
+    raise TypeError('Unable to determine a name for %r -- maybe it is not a callable?' % func)
 
 
 def obj_to_ref(obj):
     """
     Returns the path to the given object.
-    """
-    ref = '%s:%s' % (obj.__module__, obj.__name__)
-    try:
-        obj2 = ref_to_obj(ref)
-    except AttributeError:
-        pass
-    else:
-        if obj2 == obj:
-            return ref
 
-    raise ValueError('Only module level objects are supported')
+    :rtype: str
+    """
+
+    try:
+        ref = '%s:%s' % (obj.__module__, get_callable_name(obj))
+        obj2 = ref_to_obj(ref)
+        if obj != obj2:
+            raise ValueError
+    except Exception:
+        raise ValueError('Cannot determine the reference to %r' % obj)
+
+    return ref
 
 
 def ref_to_obj(ref):
     """
     Returns the object pointed to by ``ref``.
+
+    :type ref: str
     """
+
+    if not isinstance(ref, six.string_types):
+        raise TypeError('References must be strings')
+    if ':' not in ref:
+        raise ValueError('Invalid reference')
+
     modulename, rest = ref.split(':', 1)
+    try:
         obj = __import__(modulename)
+    except ImportError:
+        raise LookupError('Error resolving reference %s: could not import module' % ref)
+
+    try:
         for name in modulename.split('.')[1:] + rest.split('.'):
             obj = getattr(obj, name)
         return obj
+    except Exception:
+        raise LookupError('Error resolving reference %s: error looking up object' % ref)
 
 
 def maybe_ref(ref):
     """
-    Returns the object that the given reference points to, if it is indeed
+    Returns the object that the given reference points to, if it is indeed a reference.
-    a reference. If it is not a reference, the object is returned as-is.
+    If it is not a reference, the object is returned as-is.
     """
+
     if not isinstance(ref, str):
         return ref
     return ref_to_obj(ref)
 
 
-def to_unicode(string, encoding='ascii'):
+if six.PY2:
-    """
+    def repr_escape(string):
-    Safely converts a string to a unicode representation on any
+        if isinstance(string, six.text_type):
-    Python version.
+            return string.encode('ascii', 'backslashreplace')
-    """
-    if hasattr(string, 'decode'):
-        return string.decode(encoding, 'ignore')
         return string
+else:
+    repr_escape = lambda string: string
 
 
-if sys.version_info < (3, 0):  # pragma: nocover
+def check_callable_args(func, args, kwargs):
-    iteritems = lambda d: d.iteritems()
+    """
-    itervalues = lambda d: d.itervalues()
+    Ensures that the given callable can be called with the given arguments.
-    xrange = xrange
+
-else:  # pragma: nocover
+    :type args: tuple
-    iteritems = lambda d: d.items()
+    :type kwargs: dict
-    itervalues = lambda d: d.values()
+    """
-    xrange = range
+
+    pos_kwargs_conflicts = []  # parameters that have a match in both args and kwargs
+    positional_only_kwargs = []  # positional-only parameters that have a match in kwargs
+    unsatisfied_args = []  # parameters in signature that don't have a match in args or kwargs
+    unsatisfied_kwargs = []  # keyword-only arguments that don't have a match in kwargs
+    unmatched_args = list(args)  # args that didn't match any of the parameters in the signature
+    unmatched_kwargs = list(kwargs)  # kwargs that didn't match any of the parameters in the signature
+    has_varargs = has_var_kwargs = False  # indicates if the signature defines *args and **kwargs respectively
+
+    if signature:
+        try:
+            sig = signature(func)
+        except ValueError:
+            return  # signature() doesn't work against every kind of callable
+
+        for param in six.itervalues(sig.parameters):
+            if param.kind == param.POSITIONAL_OR_KEYWORD:
+                if param.name in unmatched_kwargs and unmatched_args:
+                    pos_kwargs_conflicts.append(param.name)
+                elif unmatched_args:
+                    del unmatched_args[0]
+                elif param.name in unmatched_kwargs:
+                    unmatched_kwargs.remove(param.name)
+                elif param.default is param.empty:
+                    unsatisfied_args.append(param.name)
+            elif param.kind == param.POSITIONAL_ONLY:
+                if unmatched_args:
+                    del unmatched_args[0]
+                elif param.name in unmatched_kwargs:
+                    unmatched_kwargs.remove(param.name)
+                    positional_only_kwargs.append(param.name)
+                elif param.default is param.empty:
+                    unsatisfied_args.append(param.name)
+            elif param.kind == param.KEYWORD_ONLY:
+                if param.name in unmatched_kwargs:
+                    unmatched_kwargs.remove(param.name)
+                elif param.default is param.empty:
+                    unsatisfied_kwargs.append(param.name)
+            elif param.kind == param.VAR_POSITIONAL:
+                has_varargs = True
+            elif param.kind == param.VAR_KEYWORD:
+                has_var_kwargs = True
+    else:
+        if not isfunction(func) and not ismethod(func) and hasattr(func, '__call__'):
+            func = func.__call__
+
+        try:
+            argspec = getargspec(func)
+        except TypeError:
+            return  # getargspec() doesn't work certain callables
+
+        argspec_args = argspec.args if not ismethod(func) else argspec.args[1:]
+        has_varargs = bool(argspec.varargs)
+        has_var_kwargs = bool(argspec.keywords)
+        for arg, default in six.moves.zip_longest(argspec_args, argspec.defaults or (), fillvalue=undefined):
+            if arg in unmatched_kwargs and unmatched_args:
+                pos_kwargs_conflicts.append(arg)
+            elif unmatched_args:
+                del unmatched_args[0]
+            elif arg in unmatched_kwargs:
+                unmatched_kwargs.remove(arg)
+            elif default is undefined:
+                unsatisfied_args.append(arg)
+
+    # Make sure there are no conflicts between args and kwargs
+    if pos_kwargs_conflicts:
+        raise ValueError('The following arguments are supplied in both args and kwargs: %s' %
+                         ', '.join(pos_kwargs_conflicts))
+
+    # Check if keyword arguments are being fed to positional-only parameters
+    if positional_only_kwargs:
+        raise ValueError('The following arguments cannot be given as keyword arguments: %s' %
+                         ', '.join(positional_only_kwargs))
+
+    # Check that the number of positional arguments minus the number of matched kwargs matches the argspec
+    if unsatisfied_args:
+        raise ValueError('The following arguments have not been supplied: %s' % ', '.join(unsatisfied_args))
+
+    # Check that all keyword-only arguments have been supplied
+    if unsatisfied_kwargs:
+        raise ValueError('The following keyword-only arguments have not been supplied in kwargs: %s' %
+                         ', '.join(unsatisfied_kwargs))
+
+    # Check that the callable can accept the given number of positional arguments
+    if not has_varargs and unmatched_args:
+        raise ValueError('The list of positional arguments is longer than the target callable can handle '
+                         '(allowed: %d, given in args: %d)' % (len(args) - len(unmatched_args), len(args)))
+
+    # Check that the callable can accept the given keyword arguments
+    if not has_var_kwargs and unmatched_kwargs:
+        raise ValueError('The target callable does not accept the following keyword arguments: %s' %
+                         ', '.join(unmatched_kwargs))

lib/beets/autotag/hooks.py

@@ -206,8 +206,8 @@ def string_dist(str1, str2):
     an edit distance, normalized by the string length, with a number of
     tweaks that reflect intuition about text.
     """
-    if str1 == None and str2 == None: return 0.0
+    if str1 is None and str2 is None: return 0.0
-    if str1 == None or str2 == None:  return 1.0
+    if str1 is None or str2 is None:  return 1.0
 
     str1 = str1.lower()
     str2 = str2.lower()

lib/beetsplug/__init__.py

@@ -0,0 +1,19 @@
+# This file is part of beets.
+# Copyright 2013, Adrian Sampson.
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+
+"""A namespace package for beets plugins."""
+
+# Make this a namespace package.
+from pkgutil import extend_path
+__path__ = extend_path(__path__, __name__)

lib/beetsplug/embedart.py

@@ -0,0 +1,194 @@
+# This file is part of beets.
+# Copyright 2014, Adrian Sampson.
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+
+"""Allows beets to embed album art into file metadata."""
+import os.path
+import logging
+import imghdr
+
+from beets.plugins import BeetsPlugin
+from beets import mediafile
+from beets import ui
+from beets.ui import decargs
+from beets.util import syspath, normpath, displayable_path
+from beets.util.artresizer import ArtResizer
+from beets import config
+
+log = logging.getLogger('beets')
+
+
+class EmbedCoverArtPlugin(BeetsPlugin):
+    """Allows albumart to be embedded into the actual files.
+    """
+    def __init__(self):
+        super(EmbedCoverArtPlugin, self).__init__()
+        self.config.add({
+            'maxwidth': 0,
+            'auto': True,
+        })
+        if self.config['maxwidth'].get(int) and \
+                not ArtResizer.shared.local:
+            self.config['maxwidth'] = 0
+            log.warn(u"embedart: ImageMagick or PIL not found; "
+                     u"'maxwidth' option ignored")
+
+    def commands(self):
+        # Embed command.
+        embed_cmd = ui.Subcommand(
+            'embedart', help='embed image files into file metadata'
+        )
+        embed_cmd.parser.add_option(
+            '-f', '--file', metavar='PATH', help='the image file to embed'
+        )
+        maxwidth = config['embedart']['maxwidth'].get(int)
+
+        def embed_func(lib, opts, args):
+            if opts.file:
+                imagepath = normpath(opts.file)
+                for item in lib.items(decargs(args)):
+                    embed_item(item, imagepath, maxwidth)
+            else:
+                for album in lib.albums(decargs(args)):
+                    embed_album(album, maxwidth)
+
+        embed_cmd.func = embed_func
+
+        # Extract command.
+        extract_cmd = ui.Subcommand('extractart',
+                                    help='extract an image from file metadata')
+        extract_cmd.parser.add_option('-o', dest='outpath',
+                                      help='image output file')
+
+        def extract_func(lib, opts, args):
+            outpath = normpath(opts.outpath or 'cover')
+            extract(lib, outpath, decargs(args))
+        extract_cmd.func = extract_func
+
+        # Clear command.
+        clear_cmd = ui.Subcommand('clearart',
+                                  help='remove images from file metadata')
+
+        def clear_func(lib, opts, args):
+            clear(lib, decargs(args))
+        clear_cmd.func = clear_func
+
+        return [embed_cmd, extract_cmd, clear_cmd]
+
+
+@EmbedCoverArtPlugin.listen('album_imported')
+def album_imported(lib, album):
+    """Automatically embed art into imported albums.
+    """
+    if album.artpath and config['embedart']['auto']:
+        embed_album(album, config['embedart']['maxwidth'].get(int))
+
+
+def embed_item(item, imagepath, maxwidth=None, itempath=None):
+    """Embed an image into the item's media file.
+    """
+    try:
+        item['images'] = [_mediafile_image(imagepath, maxwidth)]
+        item.try_write(itempath)
+    except IOError as exc:
+        log.error(u'embedart: could not read image file: {0}'.format(exc))
+    finally:
+        # We don't want to store the image in the database
+        del item['images']
+
+
+def embed_album(album, maxwidth=None):
+    """Embed album art into all of the album's items.
+    """
+    imagepath = album.artpath
+    if not imagepath:
+        log.info(u'No album art present: {0} - {1}'.
+                 format(album.albumartist, album.album))
+        return
+    if not os.path.isfile(imagepath):
+        log.error(u'Album art not found at {0}'
+                  .format(imagepath))
+        return
+
+    log.info(u'Embedding album art into {0.albumartist} - {0.album}.'
+             .format(album))
+
+    for item in album.items():
+        embed_item(item, imagepath, maxwidth)
+
+
+def _mediafile_image(image_path, maxwidth=None):
+    """Return a `mediafile.Image` object for the path.
+
+    If maxwidth is set the image is resized if necessary.
+    """
+    if maxwidth:
+        image_path = ArtResizer.shared.resize(maxwidth, syspath(image_path))
+
+    with open(syspath(image_path), 'rb') as f:
+        data = f.read()
+    return mediafile.Image(data, type=mediafile.ImageType.front)
+
+
+# 'extractart' command.
+
+def extract(lib, outpath, query):
+    item = lib.items(query).get()
+    if not item:
+        log.error(u'No item matches query.')
+        return
+
+    # Extract the art.
+    try:
+        mf = mediafile.MediaFile(syspath(item.path))
+    except mediafile.UnreadableFileError as exc:
+        log.error(u'Could not extract art from {0}: {1}'.format(
+            displayable_path(item.path), exc
+        ))
+        return
+
+    art = mf.art
+    if not art:
+        log.error(u'No album art present in {0} - {1}.'
+                  .format(item.artist, item.title))
+        return
+
+    # Add an extension to the filename.
+    ext = imghdr.what(None, h=art)
+    if not ext:
+        log.error(u'Unknown image type.')
+        return
+    outpath += '.' + ext
+
+    log.info(u'Extracting album art from: {0.artist} - {0.title}\n'
+             u'To: {1}'.format(item, displayable_path(outpath)))
+    with open(syspath(outpath), 'wb') as f:
+        f.write(art)
+
+
+# 'clearart' command.
+
+def clear(lib, query):
+    log.info(u'Clearing album art from items:')
+    for item in lib.items(query):
+        log.info(u'{0} - {1}'.format(item.artist, item.title))
+        try:
+            mf = mediafile.MediaFile(syspath(item.path),
+                                     config['id3v23'].get(bool))
+        except mediafile.UnreadableFileError as exc:
+            log.error(u'Could not clear art from {0}: {1}'.format(
+                displayable_path(item.path), exc
+            ))
+            continue
+        mf.art = None
+        mf.save()