Release v0.5.2

* master:
  Log exact write check error to log (debug)

.gitignore

@@ -6,12 +6,13 @@
 *.pyproj
 *.sln
 
-# Logs and databases #
+# Headphones files #
 ######################
 *.log
 *.db*
 *.db-journal
 *.ini
+version.lock
 logs/*
 cache/*
 
@@ -19,6 +20,7 @@ cache/*
 ##################
 *.crt
 *.key
+*.csr
 
 # OS generated files #
 ######################

.pep8

@@ -0,0 +1,17 @@
+[pep8]
+# E111 indentation is not a multiple of four
+# E121 continuation line under-indented for hanging indent
+# E122 continuation line missing indentation or outdented
+# E124 closing bracket does not match visual indentation
+# E125 continuation line with same indent as next logical line
+# E126 continuation line over-indented for hanging indent
+# E127 continuation line over-indented for visual indent
+# E128 continuation line under-indented for visual indent
+# E261 at least two spaces before inline comment
+# E262 inline comment should start with '# '
+# E265 block comment should start with '# '
+# E302 expected 2 blank lines, found 1
+# E501 line too long (312 > 160 characters)
+# E502 the backslash is redundant between brackets
+ignore = E111,E121,E122,E123,E124,E125,E126,E127,E128,E261,E262,E265,E302,E501,E502
+max-line-length = 160

.travis.yml

@@ -0,0 +1,21 @@
+# Travis CI configuration file
+# http://about.travis-ci.org/docs/
+
+language: python
+
+# Available Python versions:
+# http://about.travis-ci.org/docs/user/ci-environment/#Python-VM-images
+python:
+  - "2.6"
+  - "2.7"
+# pylint 1.4 does not run under python 2.6
+install:
+ - pip install pyOpenSSL
+ - pip install pylint==1.3.1
+ - pip install pyflakes
+ - pip install pep8
+script:
+  - pep8 headphones
+  - pylint --rcfile=pylintrc headphones
+  - pyflakes headphones
+  - nosetests headphones

API.md

@@ -0,0 +1,109 @@
+# API Reference
+The API is still pretty new and needs some serious cleaning up on the backend but should be reasonably functional. There are no error codes yet.
+
+## General structure
+The API endpoint is `http://ip:port + HTTP_ROOT + /api?apikey=$apikey&cmd=$command`
+
+Data response in JSON formatted. If executing a command like "delArtist" or "addArtist" you'll get back an "OK", else, you'll get the data you requested.
+
+## API methods
+
+### getIndex
+Fetch data from index page. Returns: ArtistName, ArtistSortName, ArtistID, Status, DateAdded, [LatestAlbum, ReleaseDate, AlbumID], HaveTracks, TotalTracks, IncludeExtras, LastUpdated, [ArtworkURL, ThumbURL]: a remote url to the artwork/thumbnail.
+
+To get the cached image path, see getArtistArt command. ThumbURL is added/updated when an artist is added/updated. If your using the database method to get the artwork, it's more reliable to use the ThumbURL than the ArtworkURL)
+
+### getArtist&id=$artistid
+Fetch artist data. returns the artist object (see above) and album info: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type, ArtworkURL: hosted image path. For cached image, see getAlbumArt command)
+
+### getAlbum&id=$albumid
+Fetch data from album page. Returns the album object, a description object and a tracks object. Tracks contain: AlbumASIN, AlbumTitle, TrackID, Format, TrackDuration (ms), ArtistName, TrackTitle, AlbumID, ArtistID, Location, TrackNumber, CleanName (stripped of punctuation /styling), BitRate)
+
+### getUpcoming
+Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type
+
+### getWanted
+Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type
+
+### getSimilar
+Returns similar artists  - with a higher "Count" being more likely to be similar. Returns: Count, ArtistName, ArtistID
+
+### getHistory
+Returns: Status, DateAdded, Title, URL (nzb), FolderName, AlbumID, Size (bytes)
+
+### getLogs
+Not working yet
+
+### findArtist&name=$artistname[&limit=$limit]
+Perform artist query on musicbrainz. Returns: url, score, name, uniquename (contains disambiguation info), id)
+
+### findAlbum&name=$albumname[&limit=$limit]
+Perform album query on musicbrainz. Returns: title, url (artist), id (artist), albumurl, albumid, score, uniquename (artist - with disambiguation)
+
+### addArtist&id=$artistid
+Add an artist to the db by artistid)
+
+### addAlbum&id=$releaseid
+Add an album to the db by album release id
+
+### delArtist&id=$artistid
+Delete artist from db by artistid)
+
+### pauseArtist&id=$artistid
+Pause an artist in db)
+
+### resumeArtist&id=$artistid
+Resume an artist in db)
+
+### refreshArtist&id=$artistid
+Refresh info for artist in db from musicbrainz
+
+### queueAlbum&id=$albumid[&new=True&lossless=True]
+Mark an album as wanted and start the searcher. Optional paramters: 'new' looks for new versions, 'lossless' looks only for lossless versions
+
+### unqueueAlbum&id=$albumid
+Unmark album as wanted / i.e. mark as skipped
+
+### forceSearch
+force search for wanted albums - not launched in a separate thread so it may take a bit to complete
+### forceProcess
+Force post process albums in download directory - also not launched in a separate thread
+
+### getVersion
+Returns some version information: git_path, install_type, current_version, installed_version, commits_behind
+
+### checkGithub
+Updates the version information above and returns getVersion data
+
+### shutdown
+Shut down headphones
+
+### restart
+Restart headphones
+
+### update
+Update headphones - you may want to check the install type in get version and not allow this if type==exe
+
+### getArtistArt&id=$artistid
+Returns either a relative path to the cached image, or a remote url if the image can't be saved to the cache dir
+
+getAlbumArt&id=$albumid
+see above
+
+### getArtistInfo&id=$artistid
+Returns Summary and Content, both formatted in html.
+
+### getAlbumInfo&id=$albumid
+See above, returns Summary and Content.
+
+### getArtistThumb&id=$artistid
+Returns either a relative path to the cached thumbnail artist image, or an http:// address if the cache dir can't be written to.
+
+### getAlbumThumb&id=$albumid
+See above.
+
+### choose_specific_download&id=$albumid
+Gives you a list of results from searcher.searchforalbum(). Basically runs a normal search, but rather than sorting them and downloading the best result, it dumps the data, which you can then pass on to download_specific_release(). Returns a list of dictionaries with params: title, size, url, provider & kind - all of these values must be passed back to download_specific_release
+
+### download_specific_release&id=albumid&title=$title&size=$size&url=$url&provider=$provider&kind=$kind
+Allows you to manually pass a choose_specific_download release back to searcher.send_to_downloader()

API_REFERENCE

@@ -1,71 +0,0 @@
-The API is still pretty new and needs some serious cleaning up on the backend but should be
-reasonably functional. There are no error codes yet, 
-
-
-General structure:
-http://localhost:8181 + HTTP_ROOT + /api?apikey=$apikey&cmd=$command
-
-Data returned in json format. If executing a command like "delArtist" or "addArtist" you'll get back an "OK", else, you'll get the data you requested
-
-$commands&parameters[&optionalparameters]:
-
-
-getIndex (fetch data from index page. Returns: ArtistName, ArtistSortName, ArtistID, Status, DateAdded,
-												[LatestAlbum, ReleaseDate, AlbumID], HaveTracks, TotalTracks,
-												IncludeExtras, LastUpdated, [ArtworkURL, ThumbURL]: a remote url to the artwork/thumbnail. To get the cached image path, see getArtistArt command.
-                                                                                                    ThumbURL is added/updated when an artist is added/updated. If your using the database method to get the artwork,
-                                                                                                    it's more reliable to use the ThumbURL than the ArtworkURL)
-
-getArtist&id=$artistid (fetch artist data. returns the artist object (see above) and album info: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type, ArtworkURL: hosted image path. For cached image, see getAlbumArt command)
-
-getAlbum&id=$albumid (fetch data from album page. Returns the album object, a description object and a tracks object. Tracks contain: AlbumASIN, AlbumTitle, TrackID, Format, TrackDuration (ms), ArtistName, TrackTitle, AlbumID, ArtistID, Location, TrackNumber, CleanName (stripped of punctuation /styling), BitRate)
-
-getUpcoming (Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type)
-
-getWanted (Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type)
-
-getSimilar (Returns similar artists  - with a higher "Count" being more likely to be similar. Returns: Count, ArtistName, ArtistID)
-
-getHistory (Returns: Status, DateAdded, Title, URL (nzb), FolderName, AlbumID, Size (bytes))
-
-getLogs (not working yet)
-
-findArtist&name=$artistname[&limit=$limit] (perform artist query on musicbrainz. Returns: url, score, name, uniquename (contains disambiguation info), id)
-
-findAlbum&name=$albumname[&limit=$limit]  (perform album query on musicbrainz. Returns: title, url (artist), id (artist), albumurl, albumid, score, uniquename (artist - with disambiguation)
-
-addArtist&id=$artistid (add an artist to the db by artistid)
-addAlbum&id=$releaseid (add an album to the db by album release id)
-
-delArtist&id=$artistid (delete artist from db by artistid)
-
-pauseArtist&id=$artistid (pause an artist in db)
-resumeArtist&id=$artistid (resume an artist in db)
-
-refreshArtist&id=$artistid (refresh info for artist in db from musicbrainz)
-
-queueAlbum&id=$albumid[&new=True&lossless=True] (Mark an album as wanted and start the searcher. Optional paramters: 'new' looks for new versions, 'lossless' looks only for lossless versions)
-unqueueAlbum&id=$albumid (Unmark album as wanted / i.e. mark as skipped)
-
-forceSearch (force search for wanted albums - not launched in a separate thread so it may take a bit to complete)
-forceProcess (force post process albums in download directory - also not launched in a separate thread)
-
-getVersion (Returns some version information: git_path, install_type, current_version, installed_version, commits_behind
-checkGithub (updates the version information above and returns getVersion data)
-
-shutdown (shut down headphones)
-restart (restart headphones)
-update (update headphones - you may want to check the install type in get version and not allow this if type==exe)
-
-getArtistArt&id=$artistid  (Returns either a relative path to the cached image, or a remote url if the image can't be saved to the cache dir)
-getAlbumArt&id=$albumid (see above)
-
-getArtistInfo&id=$artistid (Returns Summary and Content, both formatted in html)
-getAlbumInfo&id=$albumid (See above, returns Summary and Content)
-
-getArtistThumb&id=$artistid (Returns either a relative path to the cached thumbnail artist image, or an http:// address if the cache dir can't be written to)
-getAlbumThumb&id=$albumid (see above)
-
-choose_specific_download&id=$albumid (Gives you a list of results from searcher.searchforalbum(). Basically runs a normal search, but rather than sorting them and downloading the best result, it dumps the data, which you can then pass on to download_specific_release(). Returns a list of dictionaries with params: title, size, url, provider & kind - all of these values must be passed back to download_specific_release)
-
-download_specific_release&id=albumid&title=$title&size=$size&url=$url&provider=$provider&kind=$kind (Allows you to manually pass a choose_specific_download release back to searcher.send_to_downloader())

CHANGELOG.md

@@ -0,0 +1,129 @@
+# Changelog
+
+## v0.5.2
+Released 28 december 2014
+
+Highlight:
+* Added: advanced option to ignore certain folders by patterns. (#2037)
+* Added: advanced option to ignore certain files by patterns (library only)
+* Added: specify optional paths to CUE splitting tools (#1938)
+* Added: experimental support for OPB (provide details yourself!)
+* Fixed: magnet to torrent conversion (#1926)
+* Fixed: new KAT URL (#2043)
+* Fixed: LMS notifications (#1564)
+* Improved: notify user of SSL-related warnings, instead of silently failing.
+* Improved: show all search results for 'Choose Specific Release'
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.1...v0.5.2).
+
+## v0.5.1
+Released 13 November 2014
+
+Highlights:
+* Added: allow one to disable interval tasks (#2002)
+* Added: script to downgrade Headphones to last version that started (Linux only)
+* Fixed: SSL issues in CherryPy. Self-generated certificates will be 2048 now (#1995)
+* Fixed: Transmission URL detection (#1998)
+* Fixed: missing dependencies for APScheduler and CherryPy (#2001)
+* Improved: symlink infinite recursion detection
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5...v0.5.1).
+
+## v0.5
+Released 10 November 2014
+
+Highlights:
+* Added: CUE splitter
+* Added: filter search result by MusicBrainz Release Group ID
+* Added: follow symlinks while scanning library (#1953)
+* Fixed: crash during post processing (#1897)
+* Fixed: embedding lyrics (#1896)
+* Fixed: HTTP errors with older versions of Python 2.6
+* Fixed: jump back to top of page (#1948)
+* Improved: parse MusicBrainz RGID first when post processing (#1952)
+* Improved: Growl unicode characters (#1695)
+* Improved: search handling for PB and KAT
+* Improved: Last.FM API support (#1877)
+* Improved: upgraded CherryPy to version 3.6.0
+* Improved: upgraded Requests to version 2.4.1
+* Improved: upgraded APScheduler to version 3.0.1
+* Improved: lot of code refactoring
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.4...v0.5).
+
+## v0.4
+Released 20 September 2014
+
+Highlights:
+* Added: support for libav-tools (which replaces FFmpeg under Ubuntu)
+* Added: option to freeze library when post processing
+* Added: Songkick per area
+* Added: rename original NFO file (#1797)
+* Removed: removed dead search providers
+* Fixed: removed left-overs of old packages
+* Improved: rename by original folder (#1811)
+* Improved: uTorrent fixess
+* Improved: delayed loading of album art
+* Improved: search result parser
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.4...v0.4).
+
+## v0.3.4
+Released 15 May 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.3...v0.3.4).
+
+## v0.3.3
+Released 05 May 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.2...v0.3.3).
+
+## v0.3.2
+Released 17 April 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.1...v0.3.2).
+
+## v0.3.1
+Released 11 April 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3...v0.3.1).
+
+## v0.3
+Released 09 April 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2.3...v0.3).
+
+## v0.2.3
+Released 28 March 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2.2...v0.2.3).
+
+## v0.2.2
+Released 15 January 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2.1...v0.2.2).
+
+## v0.2.1
+Released 13 January 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2...v0.2.1).
+
+## v0.2
+Released 13 January 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.1.2...v0.2).
+
+## v0.1.2
+Released 11 October 2013
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.1.1...v0.1.2).
+
+## v0.1.1
+Released 26 August 2013
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.1...v0.1.1).
+
+## v0.1
+Released 05 August 2013
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/2156e1341405d07c5bcfbe994f6b354b32d94cda...v0.1).

CONTRIBUTING.md

@@ -0,0 +1,52 @@
+# Contributing to Headphones
+
+## Issues
+In case you read this because you are posting an issue, please take a minute and conside the things below. The issue tracker is not a support forum. It is primarily intended to submit bugs, improvements or feature requests. However, we are glad to help you, and make sure the problem is not caused by Headphones, but don't expect step-by-step answers.
+
+* Use the search function. Chances are that your problem is already discussed. Do not append to (closed) issues if your problem does not fit the discussion.
+* Visit the [Troubleshooting](../../wiki/TroubleShooting) wiki first.
+* Use [proper formatting](https://help.github.com/articles/github-flavored-markdown/). Paste your logs in code blocks.
+* Close your issue if you resolved it.
+
+## Pull Requests
+If you think you can contribute code to the Headphones repository, do not hesitate to submit a pull request.
+
+### Branches
+All pull requests should be based on the `develop` branch, to minimize cross merges. When you want to develop a new feature, clone the repository with `git clone origin/develop -b FEATURE_NAME`. Use meaningful commit messages.
+
+### Python Code
+
+#### Compatibility
+The code should work with Python 2.6 and 2.7. Note that Headphones runs on different platforms, including Network Attached Storage devices such as Synology.
+
+Re-use existing code. Do not hesitate to add logging in your code. You can the logger module `headphones.logger.*` for this. Web requests are invoked via `headphones.request.*` and derived ones. Use these methods to automatically add proper and meaningful error handling.
+
+#### Code conventions
+Although Headphones did not adapt a code convention in the past, we try to follow the [PEP8](http://legacy.python.org/dev/peps/pep-0008/) conventions for future code. A short summary to remind you (copied from http://wiki.ros.org/PyStyleGuide):
+
+ * 4 space indentation
+ * 80 characters per line
+ * `package_name`
+ * `ClassName`
+ * `method_name`
+ * `field_name`
+ * `_private_something`
+ * `self.__really_private_field`
+ * `_global`
+
+#### Documentation
+Document your code. Use docstrings See [PEP-257](https://www.python.org/dev/peps/pep-0257/) for more information.
+
+#### Continuous Integration
+Headphones has a configuration file for [travis-ci](https://travis-ci.org/). You can add your forked repo to Travis to have it check your code against PEP8, PyLint, and PyFlakes for you. Your pull request will show a green check mark or a red cross on each tested commit, depending on if linting passes.
+
+### HTML/Template code
+
+#### Compatibility
+HTML5 compatible browsers are targetted. There is no specific mobile version of Headphones yet.
+
+#### Conventions
+* 4 space indentation
+* `methodName`
+* `variableName`
+* `ClassName`

Headphones.py

@@ -14,15 +14,14 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os, sys
+import os
+import sys
 
 # Ensure lib added to path, before any other imports
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib/'))
 
 from headphones import webstart, logger
 
-from configobj import ConfigObj
-
 import locale
 import time
 import signal
@@ -33,6 +32,7 @@ import headphones
 signal.signal(signal.SIGINT, headphones.sig_handler)
 signal.signal(signal.SIGTERM, headphones.sig_handler)
 
+
 def main():
     """
     Headphones application entry point. Parses arguments, setups encoding and
@@ -63,16 +63,24 @@ def main():
         headphones.SYS_ENCODING = 'UTF-8'
 
     # Set up and gather command line arguments
-    parser = argparse.ArgumentParser(description='Music add-on for SABnzbd+, Transmission and more.')
+    parser = argparse.ArgumentParser(
+        description='Music add-on for SABnzbd+, Transmission and more.')
 
-    parser.add_argument('-v', '--verbose', action='store_true', help='Increase console logging verbosity')
+    parser.add_argument(
-    parser.add_argument('-q', '--quiet', action='store_true', help='Turn off console logging')
+        '-v', '--verbose', action='store_true', help='Increase console logging verbosity')
-    parser.add_argument('-d', '--daemon', action='store_true', help='Run as a daemon')
+    parser.add_argument(
-    parser.add_argument('-p', '--port', type=int, help='Force Headphones to run on a specified port')
+        '-q', '--quiet', action='store_true', help='Turn off console logging')
-    parser.add_argument('--datadir', help='Specify a directory where to store your data files')
+    parser.add_argument(
+        '-d', '--daemon', action='store_true', help='Run as a daemon')
+    parser.add_argument(
+        '-p', '--port', type=int, help='Force Headphones to run on a specified port')
+    parser.add_argument(
+        '--datadir', help='Specify a directory where to store your data files')
     parser.add_argument('--config', help='Specify a config file to use')
-    parser.add_argument('--nolaunch', action='store_true', help='Prevent browser from launching on startup')
+    parser.add_argument('--nolaunch', action='store_true',
-    parser.add_argument('--pidfile', help='Create a pid file (only relevant when running as a daemon)')
+                        help='Prevent browser from launching on startup')
+    parser.add_argument(
+        '--pidfile', help='Create a pid file (only relevant when running as a daemon)')
 
     args = parser.parse_args()
 
@@ -81,9 +89,14 @@ def main():
     if args.quiet:
         headphones.QUIET = True
 
+    # Do an intial setup of the logger.
+    logger.initLogger(console=not headphones.QUIET, log_dir=False,
+        verbose=headphones.VERBOSE)
+
     if args.daemon:
         if sys.platform == 'win32':
-            sys.stderr.write("Daemonizing not supported under Windows, starting normally\n")
+            sys.stderr.write(
+                "Daemonizing not supported under Windows, starting normally\n")
         else:
             headphones.DAEMON = True
             headphones.QUIET = True
@@ -91,11 +104,14 @@ def main():
     if args.pidfile:
         headphones.PIDFILE = str(args.pidfile)
 
-        # If the pidfile already exists, headphones may still be running, so exit
+        # If the pidfile already exists, headphones may still be running, so
+        # exit
         if os.path.exists(headphones.PIDFILE):
-            sys.exit("PID file '" + headphones.PIDFILE + "' already exists. Exiting.")
+            raise SystemExit("PID file '%s' already exists. Exiting." %
+                headphones.PIDFILE)
 
-        # The pidfile is only useful in daemon mode, make sure we can write the file properly
+        # The pidfile is only useful in daemon mode, make sure we can write the
+        # file properly
         if headphones.DAEMON:
             headphones.CREATEPID = True
 
@@ -103,9 +119,10 @@ def main():
                 with open(headphones.PIDFILE, 'w') as fp:
                     fp.write("pid\n")
             except IOError as e:
-                raise SystemExit("Unable to write PID file: %s [%d]", e.strerror, e.errno)
+                raise SystemExit("Unable to write PID file: %s", e)
         else:
-            logger.warn("Not running in daemon mode. PID file creation disabled.")
+            logger.warn("Not running in daemon mode. PID file creation " \
+                "disabled.")
 
     # Determine which data directory and config file to use
     if args.datadir:
@@ -114,27 +131,28 @@ def main():
         headphones.DATA_DIR = headphones.PROG_DIR
 
     if args.config:
-        headphones.CONFIG_FILE = args.config
+        config_file = args.config
     else:
-        headphones.CONFIG_FILE = os.path.join(headphones.DATA_DIR, 'config.ini')
+        config_file = os.path.join(headphones.DATA_DIR, 'config.ini')
 
     # Try to create the DATA_DIR if it doesn't exist
     if not os.path.exists(headphones.DATA_DIR):
         try:
             os.makedirs(headphones.DATA_DIR)
         except OSError:
-            raise SystemExit('Could not create data directory: ' + headphones.DATA_DIR + '. Exiting....')
+            raise SystemExit(
+                'Could not create data directory: ' + headphones.DATA_DIR + '. Exiting....')
 
     # Make sure the DATA_DIR is writeable
     if not os.access(headphones.DATA_DIR, os.W_OK):
-        raise SystemExit('Cannot write to the data directory: ' + headphones.DATA_DIR + '. Exiting...')
+        raise SystemExit(
+            'Cannot write to the data directory: ' + headphones.DATA_DIR + '. Exiting...')
 
     # Put the database in the DATA_DIR
     headphones.DB_FILE = os.path.join(headphones.DATA_DIR, 'headphones.db')
-    headphones.CFG = ConfigObj(headphones.CONFIG_FILE, encoding='utf-8')
 
     # Read config and start logging
-    headphones.initialize()
+    headphones.initialize(config_file)
 
     if headphones.DAEMON:
         headphones.daemonize()
@@ -147,28 +165,40 @@ def main():
         http_port = args.port
         logger.info('Using forced web server port: %i', http_port)
     else:
-        http_port = int(headphones.HTTP_PORT)
+        http_port = int(headphones.CONFIG.HTTP_PORT)
+
+    # Check if pyOpenSSL is installed. It is required for certificate generation
+    # and for CherryPy.
+    if headphones.CONFIG.ENABLE_HTTPS:
+        try:
+            import OpenSSL
+        except ImportError:
+            logger.warn("The pyOpenSSL module is missing. Install this " \
+                "module to enable HTTPS. HTTPS will be disabled.")
+            headphones.CONFIG.ENABLE_HTTPS = False
 
     # Try to start the server. Will exit here is address is already in use.
-    webstart.initialize({
+    web_config = {
         'http_port': http_port,
-        'http_host': headphones.HTTP_HOST,
+        'http_host': headphones.CONFIG.HTTP_HOST,
-        'http_root': headphones.HTTP_ROOT,
+        'http_root': headphones.CONFIG.HTTP_ROOT,
-        'http_proxy': headphones.HTTP_PROXY,
+        'http_proxy': headphones.CONFIG.HTTP_PROXY,
-        'enable_https': headphones.ENABLE_HTTPS,
+        'enable_https': headphones.CONFIG.ENABLE_HTTPS,
-        'https_cert': headphones.HTTPS_CERT,
+        'https_cert': headphones.CONFIG.HTTPS_CERT,
-        'https_key': headphones.HTTPS_KEY,
+        'https_key': headphones.CONFIG.HTTPS_KEY,
-        'http_username': headphones.HTTP_USERNAME,
+        'http_username': headphones.CONFIG.HTTP_USERNAME,
-        'http_password': headphones.HTTP_PASSWORD,
+        'http_password': headphones.CONFIG.HTTP_PASSWORD,
-    })
+    }
-
+    webstart.initialize(web_config)
-    if headphones.LAUNCH_BROWSER and not args.nolaunch:
-        headphones.launch_browser(headphones.HTTP_HOST, http_port,
-            headphones.HTTP_ROOT)
 
     # Start the background threads
     headphones.start()
 
+    # Open webbrowser
+    if headphones.CONFIG.LAUNCH_BROWSER and not args.nolaunch:
+        headphones.launch_browser(headphones.CONFIG.HTTP_HOST, http_port,
+                                  headphones.CONFIG.HTTP_ROOT)
+
     # Wait endlessy for a signal to happen
     while True:
         if not headphones.SIGNAL:
@@ -190,4 +220,4 @@ def main():
 
 # Call main()
 if __name__ == "__main__":
-    main()
+    main()

README.md

@@ -1,17 +1,18 @@
-#![preview thumb](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png)Headphones
+#![Headphones Logo](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png) Headphones
 
-###Support & Discuss
+Headphones is an automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, µTorrent and Blackhole.
 
-You are free to join the HP support community on IRC where you can ask questions, hang around and discuss anything related to HP.
+## Support & Discuss
+You are free to join the Headphones support community on IRC where you can ask questions, hang around and discuss anything related to HP.
 
-1. Use any IRC client and connect to the Freenode server.
+1. Use any IRC client and connect to the Freenode server, `irc.freenode.net`.
-2. Join #headphones
+2. Join the `#headphones` channel.
 
-###Installation and Notes
+## Installation and Notes
 
-[Read our Wiki](../../wiki) on how to install and use HeadPhones properly.
+* [Installation page](../../wiki/Installation) shows you how to install Headphones.
-
+* [Usage guide](../../wiki/Usage-guide) introduces you to Headphones.
-[**Troubleshooting** page](../../wiki/TroubleShooting) in the wiki can help you with comon problems.
+* [Troubleshooting page](../../wiki/TroubleShooting) in the wiki can help you with common problems.
 
 **Issues** can be reported on the GitHub issue tracker considering these rules:
 
@@ -27,17 +28,15 @@ You are free to join the HP support community on IRC where you can ask questions
 
 If you **comply with these rules** you can [post your request/issue](http://github.com/rembo10/headphones/issues).
 
-**Support** the project by implementing new features, solving support tickets and provide bug fixes. 
+**Support** the project by implementing new features, solving support tickets and provide bug fixes.
-If you change something in the code always make a PR to the developer branch instead of the master branch.
 
+## Screenshots
 
-###Screenshots
+Homepage (Artist Overview):
-
-Homepage (Artist Overview)
 
 ![preview thumb](http://i.imgur.com/LZO9a.png)
 
-One of the many settings pages....
+One of the many settings pages:
 
 ![preview thumb](http://i.imgur.com/xcWNy.png)
 
@@ -49,7 +48,7 @@ Import Your Favorite Artists:
 
 ![preview thumb](http://i.imgur.com/6tZoC.png)
 
-Artist Search Results (also search by album!):
+Search Results:
 
 ![preview thumb](http://i.imgur.com/rIV0P.png)
 
@@ -61,5 +60,5 @@ Album Page with track overview:
 
 ![preview thumb](http://i.imgur.com/kcjES.png)
 
-
+## License
-This is free software under the GPL v3 open source license - so feel free to do with it what you wish.
+This is free software under the GPL v3 open source license. Feel free to do with it what you wish, but any modification must be open sourced. A copy of the license is included.

contrib/clean_pyc.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# Display information
+echo "This script will remove *.pyc files. These files are generated by Python, but they can cause conflicts after an upgrade. It's safe to remove them, because they will be regenerated."
+echo "Press enter to continue, or CTRL + C to quit."
+read
+
+# Remove the *.pyc
+find "`dirname $0`/.." -type f -name "*.pyc" -exec rm -rf {} \;

contrib/downgrade.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+
+# Parameter check
+if [ -z "$1" ]; then
+    echo "Syntax: $0 <data directory>"
+    exit 1
+fi
+
+# Version file check
+if [ ! -s "$1/version.lock" ]; then
+    echo "Missing the version.lock file in the data folder, or the file is empty. Did you start Headphones at least once?"
+    exit 1
+fi
+
+# Git installation check
+if [ ! -x "$(command -v git)" ]; then
+    echo "Git is required to downgrade."
+    exit 1
+fi
+
+# Display information
+HASH=$(cat $1/version.lock)
+
+echo "This script will try to downgrade Headphones to the last version that started, version $HASH. Make sure you have a backup of your config file and database, just in case!"
+echo "Press enter to continue, or CTRL + C to quit."
+read
+
+# Downgrade
+cd "`dirname $0`/.."
+git reset --hard "$HASH"
+
+echo "All done, Headphones should be downgraded to the last version that started."

data/interfaces/default/album.html

@@ -57,13 +57,12 @@
 									search_term = ""
 								else:
 									search_term = album['SearchTerm']
-								
 							%>
 							<input type="text" value="${search_term}" name="SearchTerm" size="40" />
 						</div>
 						<input type="button" value="Save changes" onclick="doAjaxCall('editSearchTerm',$(this),'tabs',true);return false;" data-success="Search term updated"/>
 					</form>
-				</div>	
+				</div>
 			<a class="menu_link_edit" id="choose_specific_download" href="#" onclick="getAvailableDownloads()"><i class="fa fa-search"></i> Choose Specific Download</a>
 				<div id="choose_specific_download_dialog" title="Choose a specific download for this album" style="display:none" class="configtable">
 						<table class="display" id="downloads_table">
@@ -73,19 +72,20 @@
 									<th id="size">Size</th>
 									<th id="provider">Provider</th>
 									<th id="kind">Kind</th>
+									<th id="matches" title="Matches quality and/or seed settings.">Matches</th>
 								</tr>
 							</thead>
 							<tbody id="downloads_table_body">
 							</tbody>
 						</table>
-				</div>	
+				</div>
 		</div>
-	</div>	
+	</div>
 </%def>
 
 <%def name="body()">
 	<div class="table_wrapper">
-		<div id="albumheader" class="clearfix">	
+		<div id="albumheader" class="clearfix">
 			<div id="albumImg">
 				<img height="200" alt="" class="albumArt" src="artwork/album/${album['AlbumID']}">
 			</div>
@@ -231,13 +231,14 @@
 			feedback.fadeOut();
 			search_results = data
 			for( var i = 0, len = data.length; i < len; i++ ) {
-    			$('#downloads_table_body').append('<tr><td id="title"><a href="#" onclick="downloadSpecificRelease('+i+')">'+data[i].title+'</a></td><td id="size"><span title='+data[i].size+'></span>'+(data[i].size / (1024*1024)).toFixed(2)+' MB</td><td id="provider">'+data[i].provider+'</td><td id="kind">'+data[i].kind+'</td></tr>');
+    			$('#downloads_table_body').append('<tr><td id="title"><a href="#" onclick="downloadSpecificRelease('+i+')">'+data[i].title+'</a></td><td id="size"><span title='+data[i].size+'></span>'+(data[i].size / (1024*1024)).toFixed(2)+' MB</td><td id="provider">'+data[i].provider+'</td><td id="kind">'+data[i].kind+'</td><td id="matches">'+data[i].matches+'</td></tr>');
     		}
 			$('#downloads_table').dataTable({
 				"aoColumns": [
 					null,
 					{ "sType": "title-numeric"},
 					null,
+					null,
 					null
 				],
 				"aaSorting": [],

data/interfaces/default/artist.html

@@ -163,21 +163,21 @@
 	}
 
 	<%
-    if headphones.SONGKICK_FILTER_ENABLED:
+    if headphones.CONFIG.SONGKICK_FILTER_ENABLED:
         songkick_filter_enabled = "true"
     else:
         songkick_filter_enabled = "false"
 
-    if not headphones.SONGKICK_LOCATION:
+    if not headphones.CONFIG.SONGKICK_LOCATION:
     	songkick_location = "none"
     else:
-        songkick_location = headphones.SONGKICK_LOCATION
+        songkick_location = headphones.CONFIG.SONGKICK_LOCATION
-    	
+
-    if headphones.SONGKICK_ENABLED:
+    if headphones.CONFIG.SONGKICK_ENABLED:
         songkick_enabled = "true"
     else:
         songkick_enabled = "false"
-	
+
 	%>
 	function getArtistsCalendar() {
 		var template, calendarDomNode;
@@ -186,16 +186,16 @@
 
 		template = '<li><a target="_blank" href="URI"><span class="sk-name">NAME</span><span class="sk-location">LOC</span></a></li>';
 
-		$.getJSON("https://api.songkick.com/api/3.0/artists/mbid:${artist['ArtistID']}/calendar.json?apikey=${headphones.SONGKICK_APIKEY}&jsoncallback=?",
+		$.getJSON("https://api.songkick.com/api/3.0/artists/mbid:${artist['ArtistID']}/calendar.json?apikey=${headphones.CONFIG.SONGKICK_APIKEY}&jsoncallback=?",
 			function(data){
-				if (data['resultsPage'].totalEntries >= 1) {          
+				if (data['resultsPage'].totalEntries >= 1) {
-          
+
                     if( ${songkick_filter_enabled} ) {
                         data.resultsPage.results.event = $.grep(data.resultsPage.results.event, function(element,index){
                         return element.venue.metroArea.id == ${songkick_location};
                     });
                     }
-          
+
 					var tourDate;
 
 					calendarDomNode.show();
@@ -211,7 +211,7 @@
 					});
 
 					calendarDomNode.append('<li><img src="interfaces/default/images/songkick.png" alt="concerts by songkick" class="sk-logo" /></li>');
-					
+
 				$(function() {
 				  $("#artistCalendar").each(function() {
 					$("li:gt(4)", this).hide(); /* :gt() is zero-indexed */
@@ -269,7 +269,7 @@
 			$('#dialog').dialog();
 			event.preventDefault();
 		});
-		$('#menu_link_modifyextra').click(function() {
+		$('#menu_link_modifyextra').click(function(event) {
 			$('#dialog').dialog();
 			event.preventDefault();
 		});

data/interfaces/default/base.html

@@ -38,7 +38,7 @@
 			</div>
 		% elif headphones.CURRENT_VERSION != headphones.LATEST_VERSION and headphones.COMMITS_BEHIND > 0 and headphones.INSTALL_TYPE != 'win':
 			<div id="updatebar">
-				A <a href="https://github.com/${headphones.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
+				A <a href="https://github.com/${headphones.CONFIG.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
 			</div>
 		% endif
 
@@ -52,7 +52,7 @@
 				<li><a href="extras">extras</a></li>
 				<li><a href="manage">manage</a></li>
 				<li><a href="history">history</a></li>
-				<li><a href="logs" class="log">logs</a></li>
+				<li><a href="logs">logs</a></li>
 				<li><a href="config" class="config"><i class="fa fa-gear fa-lg"></i></a></li>
 			</ul>
 			<div id="searchbar">
@@ -96,8 +96,8 @@
 				%if version.HEADPHONES_VERSION != 'master':
 				(${version.HEADPHONES_VERSION})
 				%endif
-				%if headphones.GIT_BRANCH != 'master':
+				%if headphones.CONFIG.GIT_BRANCH != 'master':
-				(${headphones.GIT_BRANCH})
+				(${headphones.CONFIG.GIT_BRANCH})
 				%endif
 			</div>
 		</footer>
@@ -119,7 +119,7 @@
 	    $(document).ready(function() {
 	        $('form:first *:input[type!=hidden]:first').focus();
 	        try{
-	            var type = window.localStorage.getItem('search_type');
+	            var type = window.localStorage.getItem('search_type') || "artist";
 	            $("#search_type").val(type);
 	        } catch(e) {
 	        }
@@ -139,4 +139,4 @@
 
 <%def name="javascriptIncludes()"></%def>
 <%def name="headIncludes()"></%def>
-<%def name="headerIncludes()"></%def>
+<%def name="headerIncludes()"></%def>

data/interfaces/default/config.html

@@ -50,13 +50,13 @@
                                     <label title="Username for web server authentication. Leave empty to disable.">
                                         HTTP Username
                                     </label>
-                                    <input type="text" name="http_username" value="${config['http_user']}" size="30">
+                                    <input type="text" name="http_username" value="${config['http_username']}" size="30">
                                 </div>
                                 <div class="row">
                                     <label title="Password for web server authentication. Leave empty to disable.">
                                         HTTP Password
                                     </label>
-                                    <input type="password" name="http_password" value="${config['http_pass']}" size="30">
+                                    <input type="password" name="http_password" value="${config['http_password']}" size="30">
                                 </div>
                                 <div class="row checkbox">
                                     <input type="checkbox" name="launch_browser" value="1" ${config['launch_browser']} />
@@ -86,7 +86,7 @@
                             <fieldset>
                                 <legend>API</legend>
                                 <div class="row checkbox">
-                                    <input type="checkbox" id="useapi" name="api_enabled" id="api_enabled" value="1" ${config['api_enabled']} />
+                                    <input type="checkbox" id="api_enabled" name="api_enabled" value="1" ${config['api_enabled']} />
                                     <label title="Allow remote applications to interface with Headphones">
                                         Enable API
                                     </label>
@@ -101,6 +101,7 @@
 
                             <fieldset>
                                 <legend>Interval</legend>
+                                <small>An interval of 0 will disable a task.</small>
                                 <div class="row">
                                     <label title="Time between two searches for new downloads.">
                                         Search Interval
@@ -137,7 +138,7 @@
                     </tr>
                     <tr>
                         <div class="configmessage">
-                            <i class="fa fa-info-circle"></i> Web Interface changes require a restart to take effect
+                            <i class="fa fa-info-circle"></i> Web Interface changes require a restart to take effect. Saving settings will restart intervals.
                         </div>
                     </tr>
                 </table>
@@ -166,25 +167,25 @@
                                     <label title="SABnzbd username. Leave empty if not applicable.">
                                         SABnzbd Username
                                     </label>
-                                    <input type="text" name="sab_username" value="${config['sab_user']}" size="20">
+                                    <input type="text" name="sab_username" value="${config['sab_username']}" size="20">
                                 </div>
                                 <div class="row">
                                     <label title="SABnzbd password. Leave empty if not applicable.">
                                         SABnzbd Password
                                     </label>
-                                    <input type="password" name="sab_password" value="${config['sab_pass']}" size="20">
+                                    <input type="password" name="sab_password" value="${config['sab_password']}" size="20">
                                 </div>
                                 <div class="row">
                                     <label title="SABnzbd API key. Can be found in SABnzbd settings.">
                                         SABnzbd API key
                                     </label>
-                                    <input type="text" name="sab_apikey" value="${config['sab_api']}" size="36">
+                                    <input type="text" name="sab_apikey" value="${config['sab_apikey']}" size="36">
                                 </div>
                                 <div class="row">
                                     <label title="Name of SABnzbd category to add downloads to.">
                                         SABnzbd Category
                                     </label>
-                                    <input type="text" name="sab_category" value="${config['sab_cat']}" size="20">
+                                    <input type="text" name="sab_category" value="${config['sab_category']}" size="20">
                                 </div>
                             </fieldset>
 
@@ -200,19 +201,19 @@
                                     <label title="NZBGet username. Leave empty if not applicable">
                                         NZBget Username
                                     </label>
-                                    <input type="text" name="nzbget_username" value="${config['nzbget_user']}" size="20">
+                                    <input type="text" name="nzbget_username" value="${config['nzbget_username']}" size="20">
                                 </div>
                                 <div class="row">
                                     <label title="NZBGet password. Leave empty if not applicable">
                                         NZBget Password
                                     </label>
-                                    <input type="password" name="nzbget_password" value="${config['nzbget_pass']}" size="20">
+                                    <input type="password" name="nzbget_password" value="${config['nzbget_password']}" size="20">
                                 </div>
                                 <div class="row">
                                     <label title="Name of NZBget category to add downloads to.">
                                         NZBget Category
                                     </label>
-                                    <input type="text" name="nzbget_category" value="${config['nzbget_cat']}" size="20">
+                                    <input type="text" name="nzbget_category" value="${config['nzbget_category']}" size="20">
                                 </div>
                                 <%
                                     if config['nzbget_priority'] == -100:
@@ -266,7 +267,7 @@
                                         prio_force = ''
                                 %>
                                 <div class="row">
-                                    <label>NZBget Priority:</label>
+                                    <label>NZBget Priority</label>
                                     <select name="nzbget_priority" id="nzbget_priority">
                                         <option value="-100" ${prio_verylow}>Very Low</option>
                                         <option value="-50" ${prio_low}>Low</option>
@@ -294,7 +295,7 @@
                                         Music Download Directory:
                                     </label>
                                     <input type="text" name="download_dir" value="${config['download_dir']}" size="50">
-                                    <small>Full path where SAB or NZBget downloads your music. e.g. /Users/name/Downloads/music</small>
+                                    <small>Full path where SAB or NZBget downloads your music, e.g. /Users/name/Downloads/music</small>
                                 </div>
                                 <div class="row">
                                     <label title="Number of days of retention your usenet provider provides.">
@@ -317,10 +318,27 @@
                                     <input type="text" name="torrentblackhole_dir" value="${config['torrentblackhole_dir']}" size="50">
                                     <small>Folder your Download program watches for Torrents</small>
                                 </div>
-                                <div class="row checkbox">
+
-                                    <label>Open Magnet Links</label>
+                                <div class="row">
-                                    <input type="checkbox" name="open_magnet_links" value="1" ${config['open_magnet_links']}>
+                                    <label>Magnet links</label>
-                                    <small>Allow Headphones to open magnet links</small>
+
+                                    <label class="inline" title="Invoke shell command to open magnet URL.">
+                                        <input type="radio" name="magnet_links" id="magnet_links_0" value="0" ${config['magnet_links_0']}>
+                                        Ignore
+                                    </label>
+
+                                    <label class="inline" title="Use external service to convert magnet links into torrents.">
+                                        <input type="radio" name="magnet_links" id="magnet_links_1" value="1" ${config['magnet_links_1']}>
+                                        Open
+                                    </label>
+
+                                    <label class="inline">
+                                        <input type="radio" name="magnet_links" id="magnet_links_2" value="2" ${config['magnet_links_2']}>
+                                        Convert
+                                    </label>
+                                    <div style="clear: both"></div>
+
+                                    <small>Note: opening magnet URL's is not suitable for headless/console/terminal servers.</small>
                                 </div>
                             </fieldset>
                             <fieldset id="transmission_options">
@@ -331,11 +349,11 @@
                                 </div>
                                 <div class="row">
                                     <label>Transmission Username</label>
-                                    <input type="text" name="transmission_username" value="${config['transmission_user']}" size="30">
+                                    <input type="text" name="transmission_username" value="${config['transmission_username']}" size="30">
                                 </div>
                                 <div class="row">
                                     <label>Transmission Password</label>
-                                    <input type="password" name="transmission_password" value="${config['transmission_pass']}" size="30">
+                                    <input type="password" name="transmission_password" value="${config['transmission_password']}" size="30">
                                 </div>
                                 <div class="row">
                                     <small>Note: With Transmission, you can specify a different download directory for downloads sent from Headphones.
@@ -351,11 +369,11 @@
                                 </div>
                                 <div class="row">
                                     <label>uTorrent Username</label>
-                                    <input type="text" name="utorrent_username" value="${config['utorrent_user']}" size="30">
+                                    <input type="text" name="utorrent_username" value="${config['utorrent_username']}" size="30">
                                 </div>
                                 <div class="row">
                                     <label>uTorrent Password</label>
-                                    <input type="password" name="utorrent_password" value="${config['utorrent_pass']}" size="30">
+                                    <input type="password" name="utorrent_password" value="${config['utorrent_password']}" size="30">
                                 </div>
                                 <div class="row">
                                     <label>uTorrent Label</label>
@@ -364,14 +382,14 @@
                             </fieldset>
                             <fieldset id="general_torrent_options">
                                 <div class="row">
-                                    <label>Minimum seeders:</label>
+                                    <label>Minimum seeders</label>
                                     <input type="text" name="numberofseeders" value="${config['numberofseeders']}" size="5">
                                     <small>Number of minimum seeders a torrent must have to be accepted</small>
                                 </div>
                                 <div class="row">
                                     <label>Music Download Directory</label>
                                     <input type="text" name="download_torrent_dir" value="${config['download_torrent_dir']}" size="50">
-                                    <small>Full path where your torrent client downloads your music e.g. '/Users/name/Downloads/music'</small>
+                                    <small>Full path where your torrent client downloads your music e.g. /Users/name/Downloads/music.</small>
                                 </div>
                                 <div class="row checkbox">
                                     <label>Keep Files for Seeding</label>
@@ -382,7 +400,7 @@
                                     <input type="radio" name="prefer_torrents" id="prefer_torrents_0" value="0" ${config['prefer_torrents_0']}>NZBs
                                     <input type="radio" name="prefer_torrents" id="prefer_torrents_1" value="1" ${config['prefer_torrents_1']}>Torrents
                                     <input type="radio" name="prefer_torrents" id="prefer_torrents_2" value="2" ${config['prefer_torrents_2']}>No Preference
-                                    </div>
+                                </div>
                             </fieldset>
                         </td>
                     </tr>
@@ -397,7 +415,8 @@
                             <fieldset>
                                 <legend>Headphones Indexer</legend>
                                 <div class="row checkbox">
-                                    <input id="use_headphones_indexer" type="checkbox" name="use_headphones_indexer" onclick="initConfigCheckbox($(this));" value="1" ${config['use_headphones_indexer']} /><label>Use Headphones Indexer</label>
+                                    <input id="headphones_indexer" type="checkbox" name="headphones_indexer" onclick="initConfigCheckbox($(this));" value="1" ${config['headphones_indexer']} />
+                                    <label>Use Headphones Indexer</label>
                                 </div>
                                 <div class="config">
                                     <div class="row">
@@ -417,7 +436,7 @@
                             <fieldset>
                                 <legend>Custom Newznab Providers</legend>
                                 <div class="row checkbox">
-                                    <input  id="usenewznab" type="checkbox" name="newznab" onclick="initConfigCheckbox($(this));" value="1" ${config['use_newznab']} /><label>Use Newznab</label>
+                                    <input id="use_newznab" type="checkbox" name="use_newznab" onclick="initConfigCheckbox($(this));" value="1" ${config['use_newznab']} /><label>Use Newznab</label>
                                 </div>
                                 <div id="newznab_providers">
                                     <div class="config" id="newznab1">
@@ -428,7 +447,7 @@
                                         </div>
                                         <div class="row">
                                             <label>Newznab API</label>
-                                            <input type="text" name="newznab_apikey" value="${config['newznab_api']}" size="36">
+                                            <input type="text" name="newznab_apikey" value="${config['newznab_apikey']}" size="36">
                                         </div>
                                         <div class="row checkbox">
                                             <input id="newznab_enabled" type="checkbox" name="newznab_enabled" onclick="initConfigCheckbox($(this));" value="1" ${config['newznab_enabled']} /><label>Enabled</label>
@@ -471,7 +490,7 @@
                             <fieldset>
                                 <legend>NZBs.org</legend>
                                 <div class="row checkbox">
-                                    <input id="usenzbsorg" type="checkbox" name="nzbsorg" onclick="initConfigCheckbox($(this));" value="1" ${config['use_nzbsorg']} /><label>Use NZBs.org</label>
+                                    <input id="use_nzbsorg" type="checkbox" name="use_nzbsorg" onclick="initConfigCheckbox($(this));" value="1" ${config['use_nzbsorg']} /><label>Use NZBs.org</label>
                                 </div>
                                 <div class="config">
                                     <div class="row">
@@ -483,7 +502,7 @@
                             <fieldset>
                                 <legend>omgwtfnzbs</legend>
                                 <div class="row checkbox">
-                                    <input id="useomgwtfnzbs" type="checkbox" name="omgwtfnzbs" onclick="initConfigCheckbox($(this));" value="1" ${config['use_omgwtfnzbs']} /><label>Use omgwtfnzbs</label>
+                                    <input id="use_omgwtfnzbs" type="checkbox" name="use_omgwtfnzbs" onclick="initConfigCheckbox($(this));" value="1" ${config['use_omgwtfnzbs']} /><label>Use omgwtfnzbs</label>
                                 </div>
                                 <div class="config">
                                     <div class="row">
@@ -504,16 +523,34 @@
                                 <fieldset>
                                     <legend>The Pirate Bay</legend>
                                     <div class="row checkbox">
-                                        <input id="usepiratebay" type="checkbox" name="use_piratebay" value="1" ${config['use_piratebay']} /><label>Use The Pirate Bay</label>
+                                        <input id="use_piratebay" type="checkbox" name="use_piratebay" value="1" ${config['use_piratebay']} /><label>Use The Pirate Bay</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
-                                            <label>Proxy URL (Optional): </label>
+                                            <label>Proxy URL</label>
                                             <input type="text" name="piratebay_proxy_url" value="${config['piratebay_proxy_url']}" size="36">
+                                            <small>Optional. Leave empty for default.</small>
                                         </div>
                                         <div class="row">
-                                            <label>Seed Ratio: </label>
+                                            <label>Seed Ratio</label>
-                                            <input type="text" class="override-float" name="piratebay_ratio" value="${config['piratebay_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding">
+                                            <input type="text" class="override-float" name="piratebay_ratio" value="${config['piratebay_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding.">
+                                        </div>
+                                    </div>
+                                </fieldset>
+
+                                <fieldset>
+                                    <legend>Old Pirate Bay</legend>
+                                    <div class="row checkbox">
+                                        <input id="use_oldpiratebay" type="checkbox" name="use_oldpiratebay" value="1" ${config['use_oldpiratebay']} /><label>Use Old Pirate Bay</label>
+                                    </div>
+                                    <div class="config">
+                                        <div class="row">
+                                            <label>URL</label>
+                                            <input type="text" name="oldpiratebay_url" value="${config['oldpiratebay_url']}" size="36">
+                                        </div>
+                                        <div class="row">
+                                            <label>Seed Ratio</label>
+                                            <input type="text" class="override-float" name="oldpiratebay_ratio" value="${config['oldpiratebay_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding.">
                                         </div>
                                     </div>
                                 </fieldset>
@@ -521,15 +558,16 @@
                                 <fieldset>
                                     <legend>Kick Ass Torrents</legend>
                                     <div class="row checkbox">
-                                        <input id="usekat" type="checkbox" name="use_kat" value="1" ${config['use_kat']} /><label>Use Kick Ass Torrents</label>
+                                        <input id="use_kat" type="checkbox" name="use_kat" value="1" ${config['use_kat']} /><label>Use Kick Ass Torrents</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
-                                            <label>Proxy URL (Optional): </label>
+                                            <label>Proxy URL</label>
                                             <input type="text" name="kat_proxy_url" value="${config['kat_proxy_url']}" size="36">
+                                            <small>Optional. Leave empty for default.</small>
                                         </div>
                                         <div class="row">
-                                            <label>Seed Ratio: </label>
+                                            <label>Seed Ratio</label>
                                             <input type="text" class="override-float" name="kat_ratio" value="${config['kat_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding">
                                         </div>
                                     </div>
@@ -538,19 +576,19 @@
                                 <fieldset>
                                     <legend>Waffles.fm</legend>
                                     <div class="row checkbox">
-                                        <input id="usewaffles" type="checkbox" name="waffles" onclick="initConfigCheckbox($(this));" value="1" ${config['use_waffles']} /><label>Use Waffles.fm</label>
+                                        <input id="use_waffles" type="checkbox" name="use_waffles" onclick="initConfigCheckbox($(this));" value="1" ${config['use_waffles']} /><label>Use Waffles.fm</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
-                                            <label>UID Number: </label>
+                                            <label>UID Number</label>
                                             <input type="text" name="waffles_uid" value="${config['waffles_uid']}" size="36">
                                         </div>
                                         <div class="row">
-                                            <label>Passkey: </label>
+                                            <label>Passkey</label>
                                             <input type="password" name="waffles_passkey" value="${config['waffles_passkey']}" size="36">
                                         </div>
                                         <div class="row">
-                                            <label>Seed Ratio: </label>
+                                            <label>Seed Ratio</label>
                                             <input type="text" class="override-float" name="waffles_ratio" value="${config['waffles_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding">
                                         </div>
                                     </div>
@@ -559,19 +597,19 @@
                                 <fieldset>
                                     <legend>rutracker.org</legend>
                                     <div class="row checkbox">
-                                        <input id="userutracker" type="checkbox" name="rutracker" onclick="initConfigCheckbox($(this));" value="1" ${config['use_rutracker']} /><label>Use rutracker.org</label>
+                                        <input id="use_rutracker" type="checkbox" name="use_rutracker" onclick="initConfigCheckbox($(this));" value="1" ${config['use_rutracker']} /><label>Use rutracker.org</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
-                                            <label>User Name: </label>
+                                            <label>Userame</label>
                                             <input type="text" name="rutracker_user" value="${config['rutracker_user']}" size="36">
                                         </div>
                                         <div class="row">
-                                            <label>Password: </label>
+                                            <label>Password</label>
                                             <input type="password" name="rutracker_password" value="${config['rutracker_password']}" size="36">
                                         </div>
                                         <div class="row">
-                                            <label>Seed Ratio: </label>
+                                            <label>Seed Ratio</label>
                                             <input type="text" class="override-float" name="rutracker_ratio" value="${config['rutracker_ratio']}" size="10" title="Stop seeding when ratio met, 0 = unlimited. Scheduled job will remove torrent when post processed and finished seeding">
                                         </div>
                                     </div>
@@ -580,7 +618,7 @@
                                 <fieldset>
                                     <legend>What.cd</legend>
                                     <div class="row checkbox">
-                                        <input id="usewhatcd" type="checkbox" name="whatcd" onclick="initConfigCheckbox($(this));" value="1" ${config['use_whatcd']} /><label>Use What.cd</label>
+                                        <input id="use_whatcd" type="checkbox" name="use_whatcd" onclick="initConfigCheckbox($(this));" value="1" ${config['use_whatcd']} /><label>Use What.cd</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
@@ -601,7 +639,7 @@
                                 <fieldset>
                                     <legend>Mininova</legend>
                                      <div class="row checkbox">
-                                        <input id="usemininova" type="checkbox" name="use_mininova" value="1" ${config['use_mininova']} /><label>Use Mininova</label>
+                                        <input id="use_mininova" type="checkbox" name="use_mininova" value="1" ${config['use_mininova']} /><label>Use Mininova</label>
                                     </div>
                                     <div class="config">
                                         <div class="row">
@@ -650,15 +688,15 @@
                                     <div id="preferred_bitrate_options">
                                         <div class="row">
                                             Target bitrate
-                                            <input type="text" class="override-float" name="preferred_bitrate" value="${config['pref_bitrate']}" size="3">kbps<br>
+                                            <input type="text" class="override-float" name="preferred_bitrate" value="${config['preferred_bitrate']}" size="3">kbps<br>
                                         </div>
                                         <div class="row">
                                             <span style="padding-left: 20px">
-                                                Reject if <strong>less than</strong> <input type="text" class="override-float" name="preferred_bitrate_low_buffer" value="${config['pref_bitrate_low']}" size="3">% or <strong>more than</strong> <input type="text" class="override-float" name="preferred_bitrate_high_buffer" value="${config['pref_bitrate_high']}" size="3">% of the target size (leave blank for no limit)<br>
+                                                Reject if <strong>less than</strong> <input type="text" class="override-float" name="preferred_bitrate_low_buffer" value="${config['preferred_bitrate_low']}" size="3">% or <strong>more than</strong> <input type="text" class="override-float" name="preferred_bitrate_high_buffer" value="${config['preferred_bitrate_high']}" size="3">% of the target size (leave blank for no limit)<br>
                                             </span>
                                         </div>
                                         <div class="row checkbox left" style="padding-left: 20px">
-                                            <input type="checkbox" name="preferred_bitrate_allow_lossless" value="1" ${config['pref_bitrate_allow_lossless']}>
+                                            <input type="checkbox" name="preferred_bitrate_allow_lossless" value="1" ${config['preferred_bitrate_allow_lossless']}>
                                             <label>Allow lossless if no good lossy match found</label>
                                         </div>
                                         <div class="row checkbox left" style="padding-left: 20px">
@@ -670,7 +708,7 @@
                             </fieldset>
                             <fieldset>
                                 <legend>Search Words</legend>
-                                <small>Separate words with a comma, e.g. "word1,word2,word3"</small>
+                                <small>Separate words with a comma, e.g. "word1,word2,word3".</small>
                                 <div class="row">
                                     <label>Ignored Words</label>
                                     <input type="text" name="ignored_words" value="${config['ignored_words']}" size="50">
@@ -691,6 +729,12 @@
                         <td>
                             <fieldset>
                                 <legend>Post-Processing</legend>
+                                <div class="row checkbox left clearfix">
+                                    <label title="Use associated .cue sheet to split single file albums into multiple tracks. Requires shntool with flac or xld cli (OS X) to be installed.">
+                                        Split single file albums into multiple tracks
+                                        <input type="checkbox" name="cue_split" id="cue_split" value="1" ${config['cue_split']} />
+                                    </label>
+                                </div>
                                 <div class="row checkbox left clearfix">
                                     <label title="Freeze the database, so new artists won't be added automatically. Use this if Headphones adds artists because due to wrong snatches. This check is skipped when the folder name is appended with release group ID.">
                                         Freeze database for adding new artist
@@ -744,14 +788,18 @@
                                     </label>
                                 </div>
                                 <div class="row">
-                                    <label>Path to Destination Folder</label>
+                                    <label>
-                                    <input type="text" name="destination_dir" value="${config['dest_dir']}" size="50">
+                                        Destination Directory:
-                                    <small>e.g. /Users/name/Music/iTunes or /Volumes/share/music</small>
+                                    </label>
+                                    <input type="text" name="destination_dir" value="${config['destination_dir']}" size="50">
+                                    <small>The directory where Headphones will move file to after post processing, e.g. /Volumes/share/music.</small>
                                 </div>
                                 <div class="row">
-                                    <label>Path to Lossless Destination folder (optional)</label>
+                                    <label>
-                                    <input type="text" name="lossless_destination_dir" value="${config['lossless_dest_dir']}" size="50">
+                                        Lossless Destination Directory:
-                                    <small>Set this if you have a separate directory for lossless music</small>
+                                    </label>
+                                    <input type="text" name="lossless_destination_dir" value="${config['lossless_destination_dir']}" size="50">
+                                    <small>Optional. Set this if you have a separate directory for lossless music.</small>
                                 </div>
                             </fieldset>
                         </td>
@@ -793,7 +841,7 @@
                                         <input type="checkbox" name="prowl_onsnatch" value="1" ${config['prowl_onsnatch']} /><label>Notify on snatch?</label>
                                     </div>
                                     <div class="row">
-                                        <label>Priority (-2,-1,0,1 or 2):</label>
+                                        <label>Priority (-2,-1,0,1 or 2)</label>
                                         <input type="text" name="prowl_priority" value="${config['prowl_priority']}" size="2">
                                     </div>
                                 </div>
@@ -968,11 +1016,11 @@
                                         <input type="checkbox" name="pushover_onsnatch" value="1" ${config['pushover_onsnatch']} /><label>Notify on snatch?</label>
                                     </div>
                                     <div class="row">
-                                        <label>Priority (-1,0, or 1):</label>
+                                        <label>Priority (-1,0, or 1)</label>
                                         <input type="text" name="pushover_priority" value="${config['pushover_priority']}" size="2">
                                     </div>
                                     <div class="row">
-                                        <label>API Token (leave blank to use Headphones default):</label><input type="text" name="pushover_apitoken" value="${config['pushover_apitoken']}" size="50">
+                                        <label>API Token (leave blank to use Headphones default)</label><input type="text" name="pushover_apitoken" value="${config['pushover_apitoken']}" size="50">
                                     </div>
                                 </div>
                             </fieldset>
@@ -1140,7 +1188,7 @@
                             </div>
                             <div class="row">
                                 <div class="row">
-                                    <label>Multi-core count:</label>
+                                    <label>Multi-core count</label>
                                     <input type="text" name="encoder_multicore_count" value="${config['encoder_multicore_count']}" size="7">
                                     <small>Set equal to the number of cores, or 0 for auto</small>
                                 </div>
@@ -1241,7 +1289,7 @@
                             </div>
                             <div class="row">
                                 <label>Path to Encoder</label>
-                                <input type="text" name="encoderfolder" value="${config['encoderfolder']}" size="43">
+                                <input type="text" name="encoder_path" value="${config['encoder_path']}" size="43">
                             </div>
                         </td>
                         <td>
@@ -1278,16 +1326,21 @@
                                 <div class="row left checkbox">
                                     <input type="checkbox" name="autowant_all" value="1" ${config['autowant_all']} /><label>Automatically mark all albums as wanted</label>
                                 </div>
+                                <div class="row left checkbox">
+                                    <input type="checkbox" title="Automatically mark manually added albums from search results as wanted"
+                                           name="autowant_manually_added" value="1" ${config['autowant_manually_added']} />
+                                    <label>Automatically mark manually added albums as wanted</label>
+                                </div>
                                 <div class="row">
-                                    <label>Folder Permissions:</label>
+                                    <label>Folder Permissions</label>
                                     <input type="text" name="folder_permissions" value="${config['folder_permissions']}" size="7">
                                 </div>
                                 <div class="row">
-                                    <label>File Permissions:</label>
+                                    <label>File Permissions</label>
                                     <input type="text" name="file_permissions" value="${config['file_permissions']}" size="7">
                                 </div>
                                 <div class="row">
-                                    <label>Cache Size (in MB):</label>
+                                    <label>Cache Size (in MB)</label>
                                     <input type="text" name="cache_sizemb" value="${config['cache_sizemb']}" size="7">
                                 </div>
                             </fieldset>
@@ -1298,7 +1351,7 @@
                                     <select name="interface"><h3>
                                     %for interface in config['interface_list']:
                                     <%
-                                        if interface == headphones.INTERFACE:
+                                        if interface == headphones.CONFIG.INTERFACE:
                                             selected = 'selected="selected"'
                                         else:
                                             selected = ''
@@ -1308,11 +1361,11 @@
                                     </select>
                                 </div>
                                 <div class="row">
-                                    <label>Log Directory:</label>
+                                    <label>Log Directory</label>
                                     <input type="text" name="log_dir" value="${config['log_dir']}" size="50">
                                 </div>
                                 <div class="row">
-                                    <label>Cache Directory:</label>
+                                    <label>Cache Directory</label>
                                     <input type="text" name="cache_dir" value="${config['cache_dir']}" size="50">
                                 </div>
                             </fieldset>
@@ -1324,7 +1377,7 @@
                                 </div>
                                 <div id="songkickoptions">
                                   <div class="row">
-                                      <label>API Key:</label>
+                                      <label>API Key</label>
                                       <input type="text" name="songkick_apikey" value="${config['songkick_apikey']}" size="50">
                                   </div>
                                   <div class="row checkbox">
@@ -1332,7 +1385,7 @@
                                   </div>
                                   <div id="songkick_filteroptions">
                                     <div class="row">
-                                      <label>Metro Area ID:</label>
+                                      <label>Metro Area ID</label>
                                       <input type="text" name="songkick_location" value="${config['songkick_location']}" size="10" title="Enter the Metro Area ID, e.g. the ID for London is 24426, this can be found by clicking the link and searching/selecting the city, e.g. London should find http://www.songkick.com/metro_areas/24426-uk-london">
                                         <a target="_blank" href="http://www.songkick.com/developer/location-search">Find Area ID</a>
                                     </div>
@@ -1345,9 +1398,9 @@
                                 <div class="row">
                                     <label>Muscbrainz Mirror</label>
                                     <select name="mirror" id="mirror">
-                                    %for mirror in config['mirror_list']:
+                                    %for mirror in config['mirrorlist']:
                                         <%
-                                            if mirror == headphones.MIRROR:
+                                            if mirror == headphones.CONFIG.MIRROR:
                                                 selected = 'selected="selected"'
                                             else:
                                                 selected = ''
@@ -2008,17 +2061,18 @@
                     $( "#tabs" ).tabs();
                 });
                 initActions();
-                initConfigCheckbox("#use_headphones_indexer");
+                initConfigCheckbox("#headphones_indexer");
-                initConfigCheckbox("#usenewznab");
+                initConfigCheckbox("#use_newznab");
-                initConfigCheckbox("#usenzbsorg");
+                initConfigCheckbox("#use_nzbsorg");
-                initConfigCheckbox("#useomgwtfnzbs");
+                initConfigCheckbox("#use_omgwtfnzbs");
-				initConfigCheckbox("#usekat");
+                initConfigCheckbox("#use_kat");
-                initConfigCheckbox("#usepiratebay");
+                initConfigCheckbox("#use_piratebay");
-                initConfigCheckbox("#usemininova");
+                initConfigCheckbox("#use_oldpiratebay");
-                initConfigCheckbox("#usewaffles");
+                initConfigCheckbox("#use_mininova");
-                initConfigCheckbox("#userutracker");
+                initConfigCheckbox("#use_waffles");
-                initConfigCheckbox("#usewhatcd");
+                initConfigCheckbox("#use_rutracker");
-                initConfigCheckbox("#useapi");
+                initConfigCheckbox("#use_whatcd");
+                initConfigCheckbox("#api_enabled");
                 initConfigCheckbox("#enable_https");
 
 

data/interfaces/default/css/style.css

@@ -336,6 +336,10 @@ form .row label {
   padding-top: 7px;
   width: 175px;
 }
+form .row label.inline {
+  margin-right: 5px;
+  width: auto;
+}
 form .row input {
   margin-right: 5px;
 }

data/interfaces/default/css/style.less

@@ -191,6 +191,11 @@ form {
             line-height: normal;
             padding-top: 7px;
             width: 175px;
+
+            &.inline {
+                margin-right: 5px;
+                width: auto;
+            }
         }
         input { margin-right: 5px; }
         input[type=text], input[type=password] {

data/interfaces/default/history.html

@@ -81,8 +81,11 @@
 				"sInfoFiltered":"(filtered from _MAX_ total items)"},
 			"iDisplayLength": 25,
 			"sPaginationType": "full_numbers",
-			"aaSorting": []
+			"aaSorting": [],
-
+			"fnDrawCallback": function (o) {
+                // Jump to top of page
+                $('html,body').scrollTop(0);
+            }
 		});
 		resetFilters("history");
 	}

data/interfaces/default/index.html

@@ -10,7 +10,7 @@
 				<th id="albumart"></th>
 				<th id="name">Artist Name</th>
 				<th id="status">Status</th>
-				<th id="album">Latest Album</th>
+				<th id="album">Latest Release</th>
 				<th id="have">Have</th>
 			</tr>
 		</thead>
@@ -131,6 +131,10 @@
                     },
                 "fnInitComplete": function(oSettings, json)
                 {
+                },
+                "fnDrawCallback": function (o) {
+                    // Jump to top of page
+                    $('html,body').scrollTop(0);
                 }
             });
             $('#artist_table').on("draw.dt", function () {

data/interfaces/default/logs.html

@@ -46,51 +46,50 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-$(document).ready(function() {
+        $(document).ready(function() {
-    initActions();
+            initActions();
 
-	$('#log_table').dataTable( {
+        	$('#log_table').dataTable( {
-		"bProcessing": true,
+        		"bProcessing": true,
-        "bServerSide": true,
+                "bServerSide": true,
-		"sAjaxSource": 'getLog',
+        		"sAjaxSource": 'getLog',
-        "sPaginationType": "full_numbers",
+                "sPaginationType": "full_numbers",
-        "aaSorting": [[0, 'desc']],
+                "aaSorting": [[0, 'desc']],
-        "iDisplayLength": 25,
+                "iDisplayLength": 25,
-        "bStateSave": true,
+                "bStateSave": true,
-        "oLanguage": {
+                "oLanguage": {
-                    "sSearch":"Filter:",
+                            "sSearch":"Filter:",
-					"sLengthMenu":"Show _MENU_ lines per page",
+        					"sLengthMenu":"Show _MENU_ lines per page",
-					"sEmptyTable": "No log information available",
+        					"sEmptyTable": "No log information available",
-					"sInfo":"Showing _START_ to _END_ of _TOTAL_ lines",
+        					"sInfo":"Showing _START_ to _END_ of _TOTAL_ lines",
-					"sInfoEmpty":"Showing 0 to 0 of 0 lines",
+        					"sInfoEmpty":"Showing 0 to 0 of 0 lines",
-					"sInfoFiltered":"(filtered from _MAX_ total lines)"},
+        					"sInfoFiltered":"(filtered from _MAX_ total lines)"},
-        "fnRowCallback": function (nRow, aData, iDisplayIndex, iDisplayIndexFull) {
+                "fnRowCallback": function (nRow, aData, iDisplayIndex, iDisplayIndexFull) {
-                                if (aData[1] === "ERROR")
+                    if (aData[1] === "ERROR") {
-                                {
+                        $('td', nRow).closest('tr').addClass("gradeX");
-                                    $('td', nRow).closest('tr').addClass("gradeX");
+                    } else if (aData[1] === "WARNING") {
-                                }
+                        $('td', nRow).closest('tr').addClass("gradeW");
-                                else if (aData[1] === "WARNING")
+                    } else {
-                                {
+                        $('td', nRow).closest('tr').addClass("gradeZ");
-                                    $('td', nRow).closest('tr').addClass("gradeW");
+                    }
-                                }
-                                else
-                                {
-                                    $('td', nRow).closest('tr').addClass("gradeZ");
-                                }
 
-                                return nRow;
+                    return nRow;
-                                },
+                },
-        "fnServerData": function ( sSource, aoData, fnCallback ) {
+                "fnDrawCallback": function (o) {
-			/* Add some extra data to the sender */
+                    // Jump to top of page
-			$.getJSON( sSource, aoData, function (json) { 
+                    $('html,body').scrollTop(0);
-				fnCallback(json)
+                },
-			} );
+                "fnServerData": function ( sSource, aoData, fnCallback ) {
-		}
+        			/* Add some extra data to the sender */
-	} );
+        			$.getJSON(sSource, aoData, function (json) {
-} );
+        				fnCallback(json)
+        			});
+        		}
+        	});
+        });
 	</script>
 	<script>
-		var timer; 
+		var timer;
 		function setRefresh()
 		{
 			refreshrate = document.getElementById('refreshrate');

data/interfaces/default/manage.html

@@ -20,7 +20,7 @@
 					</div>
 				</div>
 			<a class="menu_link_edit" href="manageArtists"><i class="fa fa-pencil"></i> Manage Artists</a>
-			%if not headphones.ADD_ARTISTS:
+			%if not headphones.CONFIG.AUTO_ADD_ARTISTS:
 			<a class="menu_link_edit" href="manageNew"><i class="fa fa-pencil"></i> Manage New Artists</a>
 			%endif
 			<a class="menu_link_edit" href="manageUnmatched"><i class="fa fa-pencil"></i> Manage Unmatched</a>
@@ -53,18 +53,18 @@
 					<br/>
 					<div class="row">
 								<label for="">Path to directory</label>
-					%if headphones.MUSIC_DIR:
+					%if headphones.CONFIG.MUSIC_DIR:
-						<input type="text" value="${headphones.MUSIC_DIR}" name="path" size="70" />
+						<input type="text" value="${headphones.CONFIG.MUSIC_DIR}" name="path" size="70" />
 					%else:
 						<input type="text" value="Enter a Music Directory to scan" onfocus="if
 						(this.value==this.defaultValue) this.value='';" name="path" size="70" />
 					%endif
 					</div>
 					<div class="row checkbox"> 
-					<input type="checkbox" name="libraryscan" id="libraryscan" value="1" ${checked(headphones.LIBRARYSCAN)}><label>Automatically scan library</label>
+					<input type="checkbox" name="libraryscan" id="libraryscan" value="1" ${checked(headphones.CONFIG.LIBRARYSCAN)}><label>Automatically scan library</label>
 					</div>
 					<div class="row checkbox"> 
-					<input type="checkbox" name="autoadd" id="autoadd" value="1" ${checked(headphones.ADD_ARTISTS)}><label>Auto-add new artists</label>
+					<input type="checkbox" name="autoadd" id="autoadd" value="1" ${checked(headphones.CONFIG.AUTO_ADD_ARTISTS)}><label>Auto-add new artists</label>
 					</div>
 								
 				</fieldset>
@@ -83,8 +83,8 @@
 			<div class="row">
 				<label for="">Username</label>
 				<%
-					if headphones.LASTFM_USERNAME:
+					if headphones.CONFIG.LASTFM_USERNAME:
-						lastfmvalue = headphones.LASTFM_USERNAME
+						lastfmvalue = headphones.CONFIG.LASTFM_USERNAME
 					else:
 						lastfmvalue = ''
 				%>

data/interfaces/default/managealbums.html

@@ -112,40 +112,43 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
+		function initThisPage() {
+			$('#album_table').dataTable({
+				"bDestroy": true,
+				"aoColumns": [
+	                null,
+					null,
+					null,
+					null,
+					null,
+					null,
+					{ "sType": "title-numeric"},
+					null,
+					null
+				],
+				"aoColumnDefs": [
+				          { 'bSortable': false, 'aTargets': [ 0 ] }
+				],
+				"oLanguage": {
+					"sLengthMenu":"Show _MENU_ albums per page",
+					"sEmptyTable": "No album information available",
+					"sInfo":"Showing _TOTAL_ albums",
+					"sInfoEmpty":"Showing 0 to 0 of 0 albums",
+					"sInfoFiltered":"(filtered from _MAX_ total albums)",
+					"sSearch": ""},
+				"bPaginate": false,
+				"aaSorting": [[5, 'desc']],
+				"fnDrawCallback": function (o) {
+	                // Jump to top of page
+	                $('html,body').scrollTop(0);
+	            }
 
-	function initThisPage() {
+			});
-		$('#album_table').dataTable({
+			resetFilters("albums");
-			"bDestroy": true,
+		}
-			"aoColumns": [
+		$(document).ready(function() {
-                null,
+			initThisPage();
-				null,
-				null,
-				null,
-				null,
-				null,
-				{ "sType": "title-numeric"},
-				null,
-				null
-			],
-			"aoColumnDefs": [
-			          { 'bSortable': false, 'aTargets': [ 0 ] }
-			],				
-			"oLanguage": {
-				"sLengthMenu":"Show _MENU_ albums per page",
-				"sEmptyTable": "No album information available",
-				"sInfo":"Showing _TOTAL_ albums",
-				"sInfoEmpty":"Showing 0 to 0 of 0 albums",
-				"sInfoFiltered":"(filtered from _MAX_ total albums)",
-				"sSearch": ""},
-			"bPaginate": false,
-			"aaSorting": [[5, 'desc']]
-
 		});
-		resetFilters("albums");
-	}
-	$(document).ready(function() {
-		initThisPage();
-	});
 	</script>
 </%def>
 

data/interfaces/default/manageartists.html

@@ -86,31 +86,31 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	function initThisPage() {
+		function initThisPage() {
-		$('#artist_table').dataTable({	
+			$('#artist_table').dataTable({
-			"bDestroy":true,		
+				"bDestroy": true,
-			"aoColumns": [
+				"aoColumns": [
-				null,
+					null,
-				null,
+					null,
-				{ "sType": "title-string"},
+					{ "sType": "title-string"},
-				null,
+					null,
-				{ "sType": "title-string"},
+					{ "sType": "title-string"},
-				null
+					null
-			],
+				],
-			"oLanguage": {
+				"oLanguage": {
-				"sSearch" : "",
+					"sSearch" : "",
-				"sEmptyTable": " "
+					"sEmptyTable": " "
-			},
+				},
-			"bStateSave": true,
+				"bStateSave": true,
-			"bPaginate": false
+				"bPaginate": false
+			});
+			resetFilters("artists");
+		}
+		$(document).ready(function() {
+			initThisPage();
+		});
+		$(window).load(function(){
+			initFancybox();
 		});
-		resetFilters("artists");
-	}
-	$(document).ready(function() {
-		initThisPage();
-	});
-	$(window).load(function(){
-		initFancybox();
-	});
 	</script>
 </%def>

data/interfaces/default/managemanual.html

@@ -17,7 +17,7 @@
 <%def name="body()">
 	<div class="table_wrapper">
 	<div id="manageheader" class="title">
-		<h1 class="clearfix"><i class="fa fa-music"></i> Manage Manually Changed Albums</h1>
+		<h1 class="clearfix"><i class="fa fa-music"></i> Manage Manually Matched Albums</h1>
 	</div>
 
 	<table class="display" id="artist_table">
@@ -37,30 +37,30 @@
 				old_album_clean = album['AlbumTitle'].replace('&','%26').replace('+', '%2B').replace("'","%27")
 			%>
 			<td id="artist">${album['ArtistName']}<BR>
-				<button id="restore_artist${count_albums}" onClick="restore_Artist(this.id)">(<-) Restore Artist</button>
+				<button id="reset_artist${count_albums}" onClick="reset_Artist(this.id)">(<-) Reset Artist</button>
-				<div id="restore_artist_dialog${count_albums}" title="Restore Artist" style="display:none">
+				<div id="reset_artist_dialog${count_albums}" title="Reset Artist" style="display:none">
 					<table>
-					<tr><td>Are you sure you want to restore Local Artist: ${album['ArtistName']} to unmatched?</td></tr>
+					<tr><td>Are you sure you want to reset Local Artist: ${album['ArtistName']} to unmatched?</td></tr>
 					<tr><td align="right"><BR>
 					%if album['AlbumStatus'] == "Ignored":
-						<button href="#" onclick="doAjaxCall('markManual?action=unignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Restore Artist</button>
+						<button href="#" onclick="doAjaxCall('markManual?action=unignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully reset ${album['ArtistName']} to unmatched">Reset Artist</button>
 					%elif album['AlbumStatus'] == "Matched":
-						<button href="#" onclick="doAjaxCall('markManual?action=unmatchArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Restore Artist</button>
+						<button href="#" onclick="doAjaxCall('markManual?action=unmatchArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Reset Artist</button>
 					%endif
 					</td></tr>
 					</table>
 				</div>
 			</td>
 			<td id="album">${album['AlbumTitle']}<BR>
-				<button id="restore_album${count_albums}" onClick="restore_Album(this.id)">(<-) Restore Album</button>
+				<button id="reset_album${count_albums}" onClick="reset_Album(this.id)">(<-) Reset Album</button>
-				<div id="restore_album_dialog${count_albums}" title="Restore Album" style="display:none">
+				<div id="reset_album_dialog${count_albums}" title="Reset Album" style="display:none">
 					<table>
-					<tr><td>Are you sure you want to restore Local Album: ${album['AlbumTitle']} to unmatched?</td></tr>
+					<tr><td>Are you sure you want to reset Local Album: ${album['AlbumTitle']} to unmatched?</td></tr>
 					<tr><td align="right"><BR>
 					%if album['AlbumStatus'] == "Ignored":
-						<button href="#" onclick="doAjaxCall('markManual?action=unignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully restored ${album['AlbumTitle']} to unmatched">Restore Album</button>
+						<button href="#" onclick="doAjaxCall('markManual?action=unignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
 					%elif album['AlbumStatus'] == "Matched":
-						<button href="#" onclick="doAjaxCall('markManual?action=unmatchAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully restored ${album['AlbumTitle']} to unmatched">Restore Album</button>
+						<button href="#" onclick="doAjaxCall('markManual?action=unmatchAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
 					%endif
 					</td></tr>
 					</table>
@@ -85,35 +85,37 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	$(document).ready(function()
+	$(document).ready(function() {
-	{
+		$('#artist_table').dataTable({
-		$('#artist_table').dataTable(
+			"bStateSave": true,
-			{
+			"bPaginate": true,
-				"bStateSave": true,
+			"oLanguage": {
-				"bPaginate": true,
+				"sSearch": "",
-				"oLanguage": {
+                "sLengthMenu":"Show _MENU_ albums per page",
-					"sSearch": "",
+					"sInfo":"Showing _START_ to _END_ of _TOTAL_ albums",
-	                "sLengthMenu":"Show _MENU_ albums per page",
+					"sInfoEmpty":"Showing 0 to 0 of 0 albums",
-						"sInfo":"Showing _START_ to _END_ of _TOTAL_ albums",
+					"sInfoFiltered":"(filtered from _MAX_ total albums)",
-						"sInfoEmpty":"Showing 0 to 0 of 0 albums",
+					"sEmptyTable": " ",
-						"sInfoFiltered":"(filtered from _MAX_ total albums)",
+                    },
-						"sEmptyTable": " ",
+            "sPaginationType": "full_numbers",
-	                    },
+            "fnDrawCallback": function (o) {
-	            "sPaginationType": "full_numbers",
+                // Jump to top of page
-	         });
+                $('html,body').scrollTop(0);
+            }
+         });
 
-initActions();
+		initActions();
 	});
 
-	function restore_Artist(clicked_id) {
+	function reset_Artist(clicked_id) {
-		n=clicked_id.replace("restore_artist","");
+		n=clicked_id.replace("reset_artist","");
-		$("#restore_artist_dialog"+n).dialog();
+		$("#reset_artist_dialog"+n).dialog();
 		return false;
 	}
 
-	function restore_Album(clicked_id) {
+	function reset_Album(clicked_id) {
-		n=clicked_id.replace("restore_album","");
+		n=clicked_id.replace("reset_album","");
-		$("#restore_album_dialog"+n).dialog();
+		$("#reset_album_dialog"+n).dialog();
 		return false;
 	}
 

data/interfaces/default/managenew.html

@@ -6,7 +6,7 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a id="menu_link_scan" onclick="doAjaxCall('musicScan?path=${headphones.MUSIC_DIR}&redirect=manageNew',$(this))" data-success="Music library is getting scanned">Scan Music Library</a>
+			<a id="menu_link_scan" onclick="doAjaxCall('musicScan?path=${headphones.CONFIG.MUSIC_DIR}&redirect=manageNew',$(this))" data-success="Music library is getting scanned">Scan Music Library</a>
 		</div>		
 	</div>	
 	<a href="manage" class="back">&laquo; Back to manage overview</a>
@@ -53,18 +53,16 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	$(document).ready(function()
+	$(document).ready(function() {
-	{
+		$('#artist_table').dataTable({
-		$('#artist_table').dataTable(
+			"aaSorting": [[1, 'asc']],
-			{
+			"bStateSave": false,
-				"aaSorting": [[1, 'asc']],
+			"bPaginate": false,
-				"bStateSave": false,
+			"oLanguage": {
-				"bPaginate": false,
+				"sSearch" : ""},
-				"oLanguage": {
+		});
-					"sSearch" : ""},
+
-				
+		initActions();
-			});
-initActions();
 	});
 	</script>
 </%def>

data/interfaces/default/manageunmatched.html

@@ -118,22 +118,24 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	$(document).ready(function()
+	$(document).ready(function() {
-	{
+		$('#artist_table').dataTable({
-		$('#artist_table').dataTable(
+			"bStateSave": true,
-			{
+			"bPaginate": true,
-				"bStateSave": true,
+			"oLanguage": {
-				"bPaginate": true,
+				"sSearch": "",
-				"oLanguage": {
+                "sLengthMenu":"Show _MENU_ albums per page",
-					"sSearch": "",
+					"sInfo":"Showing _START_ to _END_ of _TOTAL_ albums",
-	                "sLengthMenu":"Show _MENU_ albums per page",
+					"sInfoEmpty":"Showing 0 to 0 of 0 albums",
-						"sInfo":"Showing _START_ to _END_ of _TOTAL_ albums",
+					"sInfoFiltered":"(filtered from _MAX_ total albums)",
-						"sInfoEmpty":"Showing 0 to 0 of 0 albums",
+					"sEmptyTable": " ",
-						"sInfoFiltered":"(filtered from _MAX_ total albums)",
+                    },
-						"sEmptyTable": " ",
+            "sPaginationType": "full_numbers",
-	                    },
+            "fnDrawCallback": function (o) {
-	            "sPaginationType": "full_numbers",
+                // Jump to top of page
-	         });
+                $('html,body').scrollTop(0);
+            }
+         });
 
 initActions();
 	});

data/interfaces/default/searchresults.html

@@ -16,6 +16,7 @@
                     <th id="tracks">Tracks</th>
                     <th id="reldate">Date</th>
                     <th id="scoresmall">Score</th>
+                    <th id="mbrelid" style="display:none;"</th>
                 %else:
                     <th id="artistname">Artist Name</th>
                     <th id="score">Score</th>
@@ -38,7 +39,7 @@
                 %>
 				<tr class="grade${grade}">
 				%if type == 'album':
-					<td id="albumart" style=" text-align: center; vertical-align: middle;"><div id="artistImg"><img title="${result['albumid']}" class="albumArt" height="50" width="50" onerror="this.onerror=null;this.src='${caa_group_url}'"></div></td>
+					<td id="albumart" style=" text-align: center; vertical-align: middle;"><div id="artistImg"><img title="${result['albumid']}" class="albumArt" height="50" width="50" onerror="tryCCA(this, '${caa_group_url}')"></div></td>
                 %else:
 					<td id="albumart"><div id="artistImg"><img title="${result['id']}" class="albumArt" height="50" width="50"></div></td>
 				%endif
@@ -50,6 +51,7 @@
                     <td id="reldate">${result['date']}</td>
                     <td id="score"><a href="${result['albumurl']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
                     <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['albumurl']}"><img src="interfaces/default/images/MusicBrainz_Album_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
+                    <td id="mbrelid" style="display:none;">${result['albumid']}</td>
                 %else:
                     <td id="artistname"><a href="addArtist?artistid=${result['id']}" title="${result['uniquename']}">${result['uniquename']}</a></td>
                     <td id="score"><a href="${result['url']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
@@ -72,7 +74,13 @@
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 
 	<script type="text/javascript">
-
+        function tryCCA(element, url) {
+            element.onerror = function() {
+                element.onerror = null;
+                element.src = "interfaces/default/images/no-cover-art.png";
+            };
+            element.src = url;
+        }
     	function getArt() {
     		$("table#searchresults_table tr td#albumart img").each(function(){
     			var id = $(this).attr('title');
@@ -85,7 +93,6 @@
     		});
     	}
     	function initThisPage() {
-            getArt();
     		$('#searchresults_table').dataTable({
 				"bDestroy": true,
 				"aoColumnDefs": [
@@ -100,11 +107,17 @@
 					"sSearch" : ""},
 				"iDisplayLength": 25,
 				"sPaginationType": "full_numbers",
-				"aaSorting": []
+				"aaSorting": [],
+                "fnDrawCallback": function (o) {
+                    // Jump to top of page
+                    $('html,body').scrollTop(0);
+                }
 			});
             $('#searchresults_table').on("draw.dt", function () {
+                getArt();
                 $("img").unveil();
             });
+            getArt();
 			resetFilters("album");
     	}
     	$(document).ready(function(){
@@ -112,4 +125,18 @@
             initThisPage();
     	});
 	</script>
+
+    <script type="text/javascript">
+        <%!
+            # Abuse JSON module for escaping JavaScript
+            import json
+        %>
+        $(document).ready(function() {
+            // Search parameter
+            $("#searchbar input[name=name]").val(${name | json.dumps});
+
+            // Album or artist
+            $("#searchbar select[name=type]").val(${type | json.dumps});
+        });
+    </script>
 </%def>

data/interfaces/default/upcoming.html

@@ -97,7 +97,7 @@
 				"oLanguage": {
 					"sEmptyTable": " "
 				},
-				"bDestroy":true,
+				"bDestroy": true,
 				"bFilter": false,
 				"bInfo": false,
 				"bPaginate": false

headphones/albumart.py

@@ -13,15 +13,18 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import request, db
+from headphones import request, db, logger
+
 
 def getAlbumArt(albumid):
     myDB = db.DBConnection()
-    asin = myDB.action('SELECT AlbumASIN from albums WHERE AlbumID=?', [albumid]).fetchone()[0]
+    asin = myDB.action(
+        'SELECT AlbumASIN from albums WHERE AlbumID=?', [albumid]).fetchone()[0]
 
     if asin:
         return 'http://ec1.images-amazon.com/images/P/%s.01.LZZZZZZZ.jpg' % asin
 
+
 def getCachedArt(albumid):
     from headphones import cache
 

headphones/albumswitcher.py

@@ -14,69 +14,84 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import headphones
-from headphones import db, logger
+from headphones import db, logger, cache
+
 
 def switch(AlbumID, ReleaseID):
-    '''
+    """
     Takes the contents from allalbums & alltracks (based on ReleaseID) and switches them into
     the albums & tracks table.
-    '''
+    """
+    logger.debug('Switching allalbums and alltracks')
     myDB = db.DBConnection()
-    oldalbumdata = myDB.action('SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
+    oldalbumdata = myDB.action(
-    newalbumdata = myDB.action('SELECT * from allalbums WHERE ReleaseID=?', [ReleaseID]).fetchone()
+        'SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
-    newtrackdata = myDB.action('SELECT * from alltracks WHERE ReleaseID=?', [ReleaseID]).fetchall()
+    newalbumdata = myDB.action(
+        'SELECT * from allalbums WHERE ReleaseID=?', [ReleaseID]).fetchone()
+    newtrackdata = myDB.action(
+        'SELECT * from alltracks WHERE ReleaseID=?', [ReleaseID]).fetchall()
     myDB.action('DELETE from tracks WHERE AlbumID=?', [AlbumID])
 
-    controlValueDict = {"AlbumID":  AlbumID}
+    controlValueDict = {"AlbumID": AlbumID}
 
-    newValueDict = {"ArtistID":         newalbumdata['ArtistID'],
+    newValueDict = {"ArtistID": newalbumdata['ArtistID'],
-                    "ArtistName":       newalbumdata['ArtistName'],
+                    "ArtistName": newalbumdata['ArtistName'],
-                    "AlbumTitle":       newalbumdata['AlbumTitle'],
+                    "AlbumTitle": newalbumdata['AlbumTitle'],
-                    "ReleaseID":        newalbumdata['ReleaseID'],
+                    "ReleaseID": newalbumdata['ReleaseID'],
-                    "AlbumASIN":        newalbumdata['AlbumASIN'],
+                    "AlbumASIN": newalbumdata['AlbumASIN'],
-                    "ReleaseDate":      newalbumdata['ReleaseDate'],
+                    "ReleaseDate": newalbumdata['ReleaseDate'],
-                    "Type":             newalbumdata['Type'],
+                    "Type": newalbumdata['Type'],
-                    "ReleaseCountry":   newalbumdata['ReleaseCountry'],
+                    "ReleaseCountry": newalbumdata['ReleaseCountry'],
-                    "ReleaseFormat":    newalbumdata['ReleaseFormat']
+                    "ReleaseFormat": newalbumdata['ReleaseFormat']
-                }
+                    }
 
     myDB.upsert("albums", newValueDict, controlValueDict)
 
+    # Update cache
+    c = cache.Cache()
+    c.remove_from_cache(AlbumID=AlbumID)
+    c.get_artwork_from_cache(AlbumID=AlbumID)
+
     for track in newtrackdata:
 
-        controlValueDict = {"TrackID":  track['TrackID'],
+        controlValueDict = {"TrackID": track['TrackID'],
-                            "AlbumID":  AlbumID}
+                            "AlbumID": AlbumID}
 
-        newValueDict = {"ArtistID":     track['ArtistID'],
+        newValueDict = {"ArtistID": track['ArtistID'],
-                    "ArtistName":       track['ArtistName'],
+                        "ArtistName": track['ArtistName'],
-                    "AlbumTitle":       track['AlbumTitle'],
+                        "AlbumTitle": track['AlbumTitle'],
-                    "AlbumASIN":        track['AlbumASIN'],
+                        "AlbumASIN": track['AlbumASIN'],
-                    "ReleaseID":        track['ReleaseID'],
+                        "ReleaseID": track['ReleaseID'],
-                    "TrackTitle":       track['TrackTitle'],
+                        "TrackTitle": track['TrackTitle'],
-                    "TrackDuration":    track['TrackDuration'],
+                        "TrackDuration": track['TrackDuration'],
-                    "TrackNumber":      track['TrackNumber'],
+                        "TrackNumber": track['TrackNumber'],
-                    "CleanName":        track['CleanName'],
+                        "CleanName": track['CleanName'],
-                    "Location":         track['Location'],
+                        "Location": track['Location'],
-                    "Format":           track['Format'],
+                        "Format": track['Format'],
-                    "BitRate":          track['BitRate']
+                        "BitRate": track['BitRate']
-                    }
+                        }
 
         myDB.upsert("tracks", newValueDict, controlValueDict)
 
-    # Mark albums as downloaded if they have at least 80% (by default, configurable) of the album
+    # Mark albums as downloaded if they have at least 80% (by default,
+    # configurable) of the album
     total_track_count = len(newtrackdata)
-    have_track_count = len(myDB.select('SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [AlbumID]))
+    have_track_count = len(myDB.select(
+        'SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [AlbumID]))
 
-    if oldalbumdata['Status'] == 'Skipped' and ((have_track_count/float(total_track_count)) >= (headphones.ALBUM_COMPLETION_PCT/100.0)):
+    if oldalbumdata['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
-        myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', AlbumID])
+        myDB.action(
+            'UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', AlbumID])
 
     # Update have track counts on index
-    totaltracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")', [newalbumdata['ArtistID']]))
+    totaltracks = len(myDB.select(
-    havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [newalbumdata['ArtistID']]))
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")', [newalbumdata['ArtistID']]))
+    havetracks = len(myDB.select(
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [newalbumdata['ArtistID']]))
 
-    controlValueDict = {"ArtistID":     newalbumdata['ArtistID']}
+    controlValueDict = {"ArtistID": newalbumdata['ArtistID']}
 
-    newValueDict = {    "TotalTracks":      totaltracks,
+    newValueDict = {"TotalTracks": totaltracks,
-                        "HaveTracks":       havetracks}
+                    "HaveTracks": havetracks}
 
     myDB.upsert("artists", newValueDict, controlValueDict)

headphones/api.py

@@ -15,18 +15,16 @@
 
 from headphones import db, mb, importer, searcher, cache, postprocessor, versioncheck, logger
 
-from xml.dom.minidom import Document
-
 import headphones
-import copy
 import json
 
-cmd_list = [ 'getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSimilar', 'getHistory', 'getLogs',
+cmd_list = ['getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSimilar', 'getHistory', 'getLogs',
             'findArtist', 'findAlbum', 'addArtist', 'delArtist', 'pauseArtist', 'resumeArtist', 'refreshArtist',
             'addAlbum', 'queueAlbum', 'unqueueAlbum', 'forceSearch', 'forceProcess', 'getVersion', 'checkGithub',
             'shutdown', 'restart', 'update', 'getArtistArt', 'getAlbumArt', 'getArtistInfo', 'getAlbumInfo',
             'getArtistThumb', 'getAlbumThumb', 'choose_specific_download', 'download_specific_release']
 
+
 class Api(object):
 
     def __init__(self):
@@ -41,16 +39,15 @@ class Api(object):
 
         self.callback = None
 
+    def checkParams(self, *args, **kwargs):
 
-    def checkParams(self,*args,**kwargs):
+        if not headphones.CONFIG.API_ENABLED:
-
-        if not headphones.API_ENABLED:
             self.data = 'API not enabled'
             return
-        if not headphones.API_KEY:
+        if not headphones.CONFIG.API_KEY:
             self.data = 'API key not generated'
             return
-        if len(headphones.API_KEY) != 32:
+        if len(headphones.CONFIG.API_KEY) != 32:
             self.data = 'API key not generated correctly'
             return
 
@@ -58,7 +55,7 @@ class Api(object):
             self.data = 'Missing api key'
             return
 
-        if kwargs['apikey'] != headphones.API_KEY:
+        if kwargs['apikey'] != headphones.CONFIG.API_KEY:
             self.data = 'Incorrect API key'
             return
         else:
@@ -82,9 +79,9 @@ class Api(object):
         if self.data == 'OK':
             logger.info('Recieved API command: %s', self.cmd)
             methodToCall = getattr(self, "_" + self.cmd)
-            result = methodToCall(**self.kwargs)
+            methodToCall(**self.kwargs)
             if 'callback' not in self.kwargs:
-                if type(self.data) == type(''):
+                if isinstance(self.data, basestring):
                     return self.data
                 else:
                     return json.dumps(self.data)
@@ -96,7 +93,7 @@ class Api(object):
         else:
             return self.data
 
-    def _dic_from_query(self,query):
+    def _dic_from_query(self, query):
 
         myDB = db.DBConnection()
         rows = myDB.select(query)
@@ -111,7 +108,8 @@ class Api(object):
 
     def _getIndex(self, **kwargs):
 
-        self.data = self._dic_from_query('SELECT * from artists order by ArtistSortName COLLATE NOCASE')
+        self.data = self._dic_from_query(
+            'SELECT * from artists order by ArtistSortName COLLATE NOCASE')
         return
 
     def _getArtist(self, **kwargs):
@@ -122,11 +120,15 @@ class Api(object):
         else:
             self.id = kwargs['id']
 
-        artist = self._dic_from_query('SELECT * from artists WHERE ArtistID="' + self.id + '"')
+        artist = self._dic_from_query(
-        albums = self._dic_from_query('SELECT * from albums WHERE ArtistID="' + self.id + '" order by ReleaseDate DESC')
+            'SELECT * from artists WHERE ArtistID="' + self.id + '"')
-        description = self._dic_from_query('SELECT * from descriptions WHERE ArtistID="' + self.id + '"')
+        albums = self._dic_from_query(
+            'SELECT * from albums WHERE ArtistID="' + self.id + '" order by ReleaseDate DESC')
+        description = self._dic_from_query(
+            'SELECT * from descriptions WHERE ArtistID="' + self.id + '"')
 
-        self.data = { 'artist': artist, 'albums': albums, 'description' : description }
+        self.data = {
+            'artist': artist, 'albums': albums, 'description': description}
         return
 
     def _getAlbum(self, **kwargs):
@@ -137,23 +139,30 @@ class Api(object):
         else:
             self.id = kwargs['id']
 
-        album = self._dic_from_query('SELECT * from albums WHERE AlbumID="' + self.id + '"')
+        album = self._dic_from_query(
-        tracks = self._dic_from_query('SELECT * from tracks WHERE AlbumID="' + self.id + '"')
+            'SELECT * from albums WHERE AlbumID="' + self.id + '"')
-        description = self._dic_from_query('SELECT * from descriptions WHERE ReleaseGroupID="' + self.id + '"')
+        tracks = self._dic_from_query(
+            'SELECT * from tracks WHERE AlbumID="' + self.id + '"')
+        description = self._dic_from_query(
+            'SELECT * from descriptions WHERE ReleaseGroupID="' + self.id + '"')
 
-        self.data = { 'album' : album, 'tracks' : tracks, 'description' : description }
+        self.data = {
+            'album': album, 'tracks': tracks, 'description': description}
         return
 
     def _getHistory(self, **kwargs):
-        self.data = self._dic_from_query('SELECT * from snatched WHERE status NOT LIKE "Seed%" order by DateAdded DESC')
+        self.data = self._dic_from_query(
+            'SELECT * from snatched WHERE status NOT LIKE "Seed%" order by DateAdded DESC')
         return
 
     def _getUpcoming(self, **kwargs):
-        self.data = self._dic_from_query("SELECT * from albums WHERE ReleaseDate > date('now') order by ReleaseDate DESC")
+        self.data = self._dic_from_query(
+            "SELECT * from albums WHERE ReleaseDate > date('now') order by ReleaseDate DESC")
         return
 
     def _getWanted(self, **kwargs):
-        self.data = self._dic_from_query("SELECT * from albums WHERE Status='Wanted'")
+        self.data = self._dic_from_query(
+            "SELECT * from albums WHERE Status='Wanted'")
         return
 
     def _getSimilar(self, **kwargs):
@@ -170,7 +179,7 @@ class Api(object):
         if 'limit' in kwargs:
             limit = kwargs['limit']
         else:
-            limit=50
+            limit = 50
 
         self.data = mb.findArtist(kwargs['name'], limit)
 
@@ -181,7 +190,7 @@ class Api(object):
         if 'limit' in kwargs:
             limit = kwargs['limit']
         else:
-            limit=50
+            limit = 50
 
         self.data = mb.findRelease(kwargs['name'], limit)
 
@@ -194,7 +203,7 @@ class Api(object):
 
         try:
             importer.addArtisttoDB(self.id)
-        except Exception, e:
+        except Exception as e:
             self.data = e
 
         return
@@ -244,7 +253,7 @@ class Api(object):
 
         try:
             importer.addArtisttoDB(self.id)
-        except Exception, e:
+        except Exception as e:
             self.data = e
 
         return
@@ -258,7 +267,7 @@ class Api(object):
 
         try:
             importer.addReleaseById(self.id)
-        except Exception, e:
+        except Exception as e:
             self.data = e
 
         return
@@ -314,11 +323,11 @@ class Api(object):
 
     def _getVersion(self, **kwargs):
         self.data = {
-            'git_path' : headphones.GIT_PATH,
+            'git_path': headphones.CONFIG.GIT_PATH,
-            'install_type' : headphones.INSTALL_TYPE,
+            'install_type': headphones.INSTALL_TYPE,
-            'current_version' : headphones.CURRENT_VERSION,
+            'current_version': headphones.CURRENT_VERSION,
-            'latest_version' : headphones.LATEST_VERSION,
+            'latest_version': headphones.LATEST_VERSION,
-            'commits_behind' : headphones.COMMITS_BEHIND,
+            'commits_behind': headphones.COMMITS_BEHIND,
         }
 
     def _checkGithub(self, **kwargs):
@@ -402,18 +411,19 @@ class Api(object):
         else:
             self.id = kwargs['id']
 
-        results = searcher.searchforalbum(self.id, choose_specific_download=True)
+        results = searcher.searchforalbum(
+            self.id, choose_specific_download=True)
 
         results_as_dicts = []
 
         for result in results:
 
             result_dict = {
-                'title':result[0],
+                'title': result[0],
-                'size':result[1],
+                'size': result[1],
-                'url':result[2],
+                'url': result[2],
-                'provider':result[3],
+                'provider': result[3],
-                'kind':result[4]
+                'kind': result[4]
             }
             results_as_dicts.append(result_dict)
 
@@ -421,7 +431,7 @@ class Api(object):
 
     def _download_specific_release(self, **kwargs):
 
-        expected_kwargs =['id', 'title','size','url','provider','kind']
+        expected_kwargs = ['id', 'title', 'size', 'url', 'provider', 'kind']
 
         for kwarg in expected_kwargs:
             if kwarg not in kwargs:
@@ -438,20 +448,24 @@ class Api(object):
         for kwarg in expected_kwargs:
             del kwargs[kwarg]
 
-        # Handle situations where the torrent url contains arguments that are parsed
+        # Handle situations where the torrent url contains arguments that are
+        # parsed
         if kwargs:
-            import urllib, urllib2
+            import urllib
-            url = urllib2.quote(url, safe=":?/=&") + '&' + urllib.urlencode(kwargs)
+            import urllib2
+            url = urllib2.quote(
+                url, safe=":?/=&") + '&' + urllib.urlencode(kwargs)
 
         try:
-            result = [(title,int(size),url,provider,kind)]
+            result = [(title, int(size), url, provider, kind)]
         except ValueError:
-            result = [(title,float(size),url,provider,kind)]
+            result = [(title, float(size), url, provider, kind)]
 
         logger.info(u"Making sure we can download the chosen result")
         (data, bestqual) = searcher.preprocess(result)
 
         if data and bestqual:
-          myDB = db.DBConnection()
+            myDB = db.DBConnection()
-          album = myDB.action('SELECT * from albums WHERE AlbumID=?', [id]).fetchone()
+            album = myDB.action(
-          searcher.send_to_downloader(data, bestqual, album)
+                'SELECT * from albums WHERE AlbumID=?', [id]).fetchone()
+            searcher.send_to_downloader(data, bestqual, album)

headphones/cache.py

@@ -14,14 +14,13 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import glob
-import urllib
 import headphones
 
 from headphones import db, helpers, logger, lastfm, request
 
 LASTFM_API_KEY = "690e1ed3bc00bc91804cd8f7fe5ed6d4"
 
+
 class Cache(object):
     """
     This class deals with getting, storing and serving up artwork (album
@@ -40,7 +39,7 @@ class Cache(object):
     and for info it is <musicbrainzid>.<date>.txt
     """
 
-    path_to_art_cache = os.path.join(headphones.CACHE_DIR, 'artwork')
+    path_to_art_cache = os.path.join(headphones.CONFIG.CACHE_DIR, 'artwork')
 
     def __init__(self):
         self.id = None
@@ -59,12 +58,12 @@ class Cache(object):
         self.info_summary = None
         self.info_content = None
 
-    def _findfilesstartingwith(self,pattern,folder):
+    def _findfilesstartingwith(self, pattern, folder):
         files = []
         if os.path.exists(folder):
             for fname in os.listdir(folder):
                 if fname.startswith(pattern):
-                    files.append(os.path.join(folder,fname))
+                    files.append(os.path.join(folder, fname))
         return files
 
     def _exists(self, type):
@@ -72,14 +71,14 @@ class Cache(object):
         self.thumb_files = []
 
         if type == 'artwork':
-            self.artwork_files = self._findfilesstartingwith(self.id,self.path_to_art_cache)
+            self.artwork_files = self._findfilesstartingwith(self.id, self.path_to_art_cache)
             if self.artwork_files:
                 return True
             else:
                 return False
 
         elif type == 'thumb':
-            self.thumb_files = self._findfilesstartingwith("T_" + self.id,self.path_to_art_cache)
+            self.thumb_files = self._findfilesstartingwith("T_" + self.id, self.path_to_art_cache)
             if self.thumb_files:
                 return True
             else:
@@ -88,11 +87,10 @@ class Cache(object):
     def _get_age(self, date):
         # There's probably a better way to do this
         split_date = date.split('-')
-        days_old = int(split_date[0])*365 + int(split_date[1])*30 + int(split_date[2])
+        days_old = int(split_date[0]) * 365 + int(split_date[1]) * 30 + int(split_date[2])
 
         return days_old
 
-
     def _is_current(self, filename=None, date=None):
 
         if filename:
@@ -191,11 +189,11 @@ class Cache(object):
         if not db_info or not db_info['LastUpdated'] or not self._is_current(date=db_info['LastUpdated']):
 
             self._update_cache()
-            info_dict = { 'Summary' : self.info_summary, 'Content' : self.info_content }
+            info_dict = {'Summary': self.info_summary, 'Content': self.info_content}
             return info_dict
 
         else:
-            info_dict = { 'Summary' : db_info['Summary'], 'Content' : db_info['Content'] }
+            info_dict = {'Summary': db_info['Summary'], 'Content': db_info['Content']}
             return info_dict
 
     def get_image_links(self, ArtistID=None, AlbumID=None):
@@ -240,7 +238,37 @@ class Cache(object):
             if not thumb_url:
                 logger.debug('No album thumbnail image found on last.fm')
 
-        return {'artwork' : image_url, 'thumbnail' : thumb_url }
+        return {'artwork': image_url, 'thumbnail': thumb_url}
+
+    def remove_from_cache(self, ArtistID=None, AlbumID=None):
+        """
+        Pass a musicbrainz id to this function (either ArtistID or AlbumID)
+        """
+
+        if ArtistID:
+            self.id = ArtistID
+            self.id_type = 'artist'
+        else:
+            self.id = AlbumID
+            self.id_type = 'album'
+
+        self.query_type = 'artwork'
+
+        if self._exists('artwork'):
+            for artwork_file in self.artwork_files:
+                try:
+                    os.remove(artwork_file)
+                except:
+                    logger.warn('Error deleting file from the cache: %s', artwork_file)
+
+        self.query_type = 'thumb'
+
+        if self._exists('thumb'):
+            for thumb_file in self.thumb_files:
+                try:
+                    os.remove(thumb_file)
+                except Exception:
+                    logger.warn('Error deleting file from the cache: %s', thumb_file)
 
     def _update_cache(self):
         '''
@@ -249,6 +277,7 @@ class Cache(object):
         myDB = db.DBConnection()
 
         # Since lastfm uses release ids rather than release group ids for albums, we have to do a artist + album search for albums
+        # Exception is when adding albums manually, then we should use release id
         if self.id_type == 'artist':
 
             data = lastfm.request_lastfm("artist.getinfo", mbid=self.id, api_key=LASTFM_API_KEY)
@@ -278,8 +307,13 @@ class Cache(object):
 
         else:
 
-            dbartist = myDB.action('SELECT ArtistName, AlbumTitle FROM albums WHERE AlbumID=?', [self.id]).fetchone()
+            dbalbum = myDB.action('SELECT ArtistName, AlbumTitle, ReleaseID FROM albums WHERE AlbumID=?', [self.id]).fetchone()
-            data = lastfm.request_lastfm("album.getinfo", artist=dbartist['ArtistName'], album=dbartist['AlbumTitle'], api_key=LASTFM_API_KEY)
+            if dbalbum['ReleaseID'] != self.id:
+                data = lastfm.request_lastfm("album.getinfo", mbid=dbalbum['ReleaseID'], api_key=LASTFM_API_KEY)
+                if not data:
+                    data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'], album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)
+            else:
+                data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'], album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)
 
             if not data:
                 return
@@ -307,13 +341,13 @@ class Cache(object):
 
         #Save the content & summary to the database no matter what if we've opened up the url
         if self.id_type == 'artist':
-            controlValueDict = {"ArtistID":     self.id}
+            controlValueDict = {"ArtistID": self.id}
         else:
-            controlValueDict = {"ReleaseGroupID":     self.id}
+            controlValueDict = {"ReleaseGroupID": self.id}
 
-        newValueDict = {"Summary":       self.info_summary,
+        newValueDict = {"Summary": self.info_summary,
-                        "Content":       self.info_content,
+                        "Content": self.info_content,
-                        "LastUpdated":   helpers.today()}
+                        "LastUpdated": helpers.today()}
 
         myDB.upsert("descriptions", newValueDict, controlValueDict)
 
@@ -340,7 +374,7 @@ class Cache(object):
                 if not os.path.isdir(self.path_to_art_cache):
                     try:
                         os.makedirs(self.path_to_art_cache)
-                    except Exception, e:
+                    except Exception as e:
                         logger.error('Unable to create artwork cache dir. Error: %s', e)
                         self.artwork_errors = True
                         self.artwork_url = image_url
@@ -364,7 +398,7 @@ class Cache(object):
                     self.artwork_url = image_url
 
         # Grab the thumbnail as well if we're getting the full artwork (as long as it's missing/outdated
-        if thumb_url and self.query_type in ['thumb','artwork'] and not (self.thumb_files and self._is_current(self.thumb_files[0])):
+        if thumb_url and self.query_type in ['thumb', 'artwork'] and not (self.thumb_files and self._is_current(self.thumb_files[0])):
             artwork = request.request_content(thumb_url, timeout=20)
 
             if artwork:
@@ -395,6 +429,7 @@ class Cache(object):
                     self.thumb_errors = True
                     self.thumb_url = image_url
 
+
 def getArtwork(ArtistID=None, AlbumID=None):
 
     c = Cache()
@@ -409,6 +444,7 @@ def getArtwork(ArtistID=None, AlbumID=None):
         artwork_file = os.path.basename(artwork_path)
         return "cache/artwork/" + artwork_file
 
+
 def getThumb(ArtistID=None, AlbumID=None):
 
     c = Cache()
@@ -423,6 +459,7 @@ def getThumb(ArtistID=None, AlbumID=None):
         thumbnail_file = os.path.basename(artwork_path)
         return "cache/artwork/" + thumbnail_file
 
+
 def getInfo(ArtistID=None, AlbumID=None):
 
     c = Cache()
@@ -431,6 +468,7 @@ def getInfo(ArtistID=None, AlbumID=None):
 
     return info_dict
 
+
 def getImageLinks(ArtistID=None, AlbumID=None):
 
     c = Cache()

headphones/classes.py

@@ -17,31 +17,32 @@
 ## Stolen from Sick-Beard's classes.py ##
 #########################################
 
-import headphones
 
 import urllib
-import datetime
 
 from common import USER_AGENT
 
+
 class HeadphonesURLopener(urllib.FancyURLopener):
     version = USER_AGENT
 
+
 class AuthURLOpener(HeadphonesURLopener):
     """
     URLOpener class that supports http auth without needing interactive password entry.
     If the provided username/password don't work it simply fails.
-    
+
     user: username to use for HTTP auth
     pw: password to use for HTTP auth
     """
+
     def __init__(self, user, pw):
         self.username = user
         self.password = pw
 
         # remember if we've tried the username/password before
         self.numTries = 0
-        
+
         # call the base class
         urllib.FancyURLopener.__init__(self)
 
@@ -55,7 +56,7 @@ class AuthURLOpener(HeadphonesURLopener):
         if self.numTries == 0:
             self.numTries = 1
             return (self.username, self.password)
-        
+
         # if we've tried before then return blank which cancels the request
         else:
             return ('', '')
@@ -65,6 +66,7 @@ class AuthURLOpener(HeadphonesURLopener):
         self.numTries = 0
         return HeadphonesURLopener.open(self, url)
 
+
 class SearchResult:
     """
     Represents a search result from an indexer.
@@ -87,7 +89,7 @@ class SearchResult:
 
     def __str__(self):
 
-        if self.provider == None:
+        if self.provider is None:
             return "Invalid provider, unable to print self"
 
         myString = self.provider.name + " @ " + self.url + "\n"
@@ -96,24 +98,28 @@ class SearchResult:
             myString += "  " + extra + "\n"
         return myString
 
+
 class NZBSearchResult(SearchResult):
     """
     Regular NZB result with an URL to the NZB
     """
     resultType = "nzb"
 
+
 class NZBDataSearchResult(SearchResult):
     """
     NZB result where the actual NZB XML data is stored in the extraInfo
     """
     resultType = "nzbdata"
 
+
 class TorrentSearchResult(SearchResult):
     """
     Torrent result with an URL to the torrent
     """
     resultType = "torrent"
 
+
 class Proper:
     def __init__(self, name, url, date):
         self.name = name
@@ -127,4 +133,4 @@ class Proper:
         self.episode = -1
 
     def __str__(self):
-        return str(self.date)+" "+self.name+" "+str(self.season)+"x"+str(self.episode)+" of "+str(self.tvdbid)
+        return str(self.date) + " " + self.name + " " + str(self.season) + "x" + str(self.episode) + " of " + str(self.tvdbid)

headphones/common.py

@@ -18,12 +18,15 @@ Created on Aug 1, 2011
 
 @author: Michael
 '''
-import platform, operator, os, re
+import platform
+import operator
+import os
+import re
 
 from headphones import version
 
 #Identify Our Application
-USER_AGENT = 'Headphones/-'+version.HEADPHONES_VERSION+' ('+platform.system()+' '+platform.release()+')'
+USER_AGENT = 'Headphones/-' + version.HEADPHONES_VERSION + ' (' + platform.system() + ' ' + platform.release() + ')'
 
 ### Notification Types
 NOTIFY_SNATCH = 1
@@ -44,17 +47,18 @@ ARCHIVED = 6 # releases that you don't have locally (counts toward download comp
 IGNORED = 7 # releases that you don't want included in your download stats
 SNATCHED_PROPER = 9 # qualified with quality
 
+
 class Quality:
 
     NONE = 0
-    B192 = 1<<1     # 2
+    B192 = 1 << 1     # 2
-    VBR =  1<<2     # 4
+    VBR = 1 << 2     # 4
-    B256 = 1<<3     # 8
+    B256 = 1 << 3     # 8
-    B320 = 1<<4     #16
+    B320 = 1 << 4     #16
-    FLAC = 1<<5     #32
+    FLAC = 1 << 5     #32
 
     # put these bits at the other end of the spectrum, far enough out that they shouldn't interfere
-    UNKNOWN = 1<<15
+    UNKNOWN = 1 << 15
 
     qualityStrings = {NONE: "N/A",
                       UNKNOWN: "Unknown",
@@ -71,7 +75,7 @@ class Quality:
     def _getStatusStrings(status):
         toReturn = {}
         for x in Quality.qualityStrings.keys():
-            toReturn[Quality.compositeStatus(status, x)] = Quality.statusPrefixes[status]+" ("+Quality.qualityStrings[x]+")"
+            toReturn[Quality.compositeStatus(status, x)] = Quality.statusPrefixes[status] + " (" + Quality.qualityStrings[x] + ")"
         return toReturn
 
     @staticmethod
@@ -82,7 +86,7 @@ class Quality:
             anyQuality = reduce(operator.or_, anyQualities)
         if bestQualities:
             bestQuality = reduce(operator.or_, bestQualities)
-        return anyQuality | (bestQuality<<16)
+        return anyQuality | (bestQuality << 16)
 
     @staticmethod
     def splitQuality(quality):
@@ -91,7 +95,7 @@ class Quality:
         for curQual in Quality.qualityStrings.keys():
             if curQual & quality:
                 anyQualities.append(curQual)
-            if curQual<<16 & quality:
+            if curQual << 16 & quality:
                 bestQualities.append(curQual)
 
         return (anyQualities, bestQualities)
@@ -106,7 +110,7 @@ class Quality:
             if x == Quality.UNKNOWN:
                 continue
 
-            regex = '\W'+Quality.qualityStrings[x].replace(' ','\W')+'\W'
+            regex = '\W' + Quality.qualityStrings[x].replace(' ', '\W') + '\W'
             regex_match = re.search(regex, name, re.I)
             if regex_match:
                 return x
@@ -147,8 +151,8 @@ class Quality:
     def splitCompositeStatus(status):
         """Returns a tuple containing (status, quality)"""
         for x in sorted(Quality.qualityStrings.keys(), reverse=True):
-            if status > x*100:
+            if status > x * 100:
-                return (status-x*100, x)
+                return (status - x * 100, x)
 
         return (Quality.NONE, status)
 

headphones/config.py

@@ -0,0 +1,441 @@
+import headphones.logger
+import itertools
+import os
+import re
+from configobj import ConfigObj
+
+
+def bool_int(value):
+    """
+    Casts a config value into a 0 or 1
+    """
+    if isinstance(value, basestring):
+        if value.lower() in ('', '0', 'false', 'f', 'no', 'n', 'off'):
+            value = 0
+    return int(bool(value))
+
+_CONFIG_DEFINITIONS = {
+    'ADD_ALBUM_ART': (int, 'General', 0),
+    'ADVANCEDENCODER': (str, 'General', ''),
+    'ALBUM_ART_FORMAT': (str, 'General', 'folder'),
+    # This is used in importer.py to determine how complete an album needs to
+    # be - to be considered "downloaded". Percentage from 0-100
+    'ALBUM_COMPLETION_PCT': (int, 'Advanced', 80),
+    'API_ENABLED': (int, 'General', 0),
+    'API_KEY': (str, 'General', ''),
+    'AUTOWANT_ALL': (int, 'General', 0),
+    'AUTOWANT_MANUALLY_ADDED': (int, 'General', 1),
+    'AUTOWANT_UPCOMING': (int, 'General', 1),
+    'AUTO_ADD_ARTISTS': (int, 'General', 1),
+    'BITRATE': (int, 'General', 192),
+    'BLACKHOLE': (int, 'General', 0),
+    'BLACKHOLE_DIR': (str, 'General', ''),
+    'BOXCAR_ENABLED': (int, 'Boxcar', 0),
+    'BOXCAR_ONSNATCH': (int, 'Boxcar', 0),
+    'BOXCAR_TOKEN': (str, 'Boxcar', ''),
+    'CACHE_DIR': (str, 'General', ''),
+    'CACHE_SIZEMB': (int, 'Advanced', 32),
+    'CHECK_GITHUB': (int, 'General', 1),
+    'CHECK_GITHUB_INTERVAL': (int, 'General', 360),
+    'CHECK_GITHUB_ON_STARTUP': (int, 'General', 1),
+    'CLEANUP_FILES': (int, 'General', 0),
+    'CONFIG_VERSION': (str, 'General', '0'),
+    'CORRECT_METADATA': (int, 'General', 0),
+    'CUE_SPLIT': (int, 'General', 1),
+    'CUE_SPLIT_FLAC_PATH': (str, 'General', ''),
+    'CUE_SPLIT_SHNTOOL_PATH': (str, 'General', ''),
+    'CUSTOMHOST': (str, 'General', 'localhost'),
+    'CUSTOMPORT': (int, 'General', 5000),
+    'CUSTOMSLEEP': (int, 'General', 1),
+    'DELETE_LOSSLESS_FILES': (int, 'General', 1),
+    'DESTINATION_DIR': (str, 'General', ''),
+    'DETECT_BITRATE': (int, 'General', 0),
+    'DOWNLOAD_DIR': (str, 'General', ''),
+    'DOWNLOAD_SCAN_INTERVAL': (int, 'General', 5),
+    'DOWNLOAD_TORRENT_DIR': (str, 'General', ''),
+    'DO_NOT_OVERRIDE_GIT_BRANCH': (int, 'General', 0),
+    'EMBED_ALBUM_ART': (int, 'General', 0),
+    'EMBED_LYRICS': (int, 'General', 0),
+    'ENABLE_HTTPS': (int, 'General', 0),
+    'ENCODER': (str, 'General', 'ffmpeg'),
+    'ENCODERFOLDER': (str, 'General', ''),
+    'ENCODERLOSSLESS': (int, 'General', 1),
+    'ENCODEROUTPUTFORMAT': (str, 'General', 'mp3'),
+    'ENCODERQUALITY': (int, 'General', 2),
+    'ENCODERVBRCBR': (str, 'General', 'cbr'),
+    'ENCODER_MULTICORE': (int, 'General', 0),
+    'ENCODER_MULTICORE_COUNT': (int, 'General', 0),
+    'ENCODER_PATH': (str, 'General', ''),
+    'EXTRAS': (str, 'General', ''),
+    'EXTRA_NEWZNABS': (list, 'Newznab', ''),
+    'FILE_FORMAT': (str, 'General', 'Track Artist - Album [Year] - Title'),
+    'FILE_PERMISSIONS': (str, 'General', '0644'),
+    'FILE_UNDERSCORES': (int, 'General', 0),
+    'FOLDER_FORMAT': (str, 'General', 'Artist/Album [Year]'),
+    'FOLDER_PERMISSIONS': (str, 'General', '0755'),
+    'FREEZE_DB': (int, 'General', 0),
+    'GIT_BRANCH': (str, 'General', 'master'),
+    'GIT_PATH': (str, 'General', ''),
+    'GIT_USER': (str, 'General', 'rembo10'),
+    'GROWL_ENABLED': (int, 'Growl', 0),
+    'GROWL_HOST': (str, 'Growl', ''),
+    'GROWL_ONSNATCH': (int, 'Growl', 0),
+    'GROWL_PASSWORD': (str, 'Growl', ''),
+    'HEADPHONES_INDEXER': (bool_int, 'General', False),
+    'HPPASS': (str, 'General', ''),
+    'HPUSER': (str, 'General', ''),
+    'HTTPS_CERT': (str, 'General', ''),
+    'HTTPS_KEY': (str, 'General', ''),
+    'HTTP_HOST': (str, 'General', '0.0.0.0'),
+    'HTTP_PASSWORD': (str, 'General', ''),
+    'HTTP_PORT': (int, 'General', 8181),
+    'HTTP_PROXY': (int, 'General', 0),
+    'HTTP_ROOT': (str, 'General', '/'),
+    'HTTP_USERNAME': (str, 'General', ''),
+    'IGNORED_WORDS': (str, 'General', ''),
+    'IGNORED_FOLDERS': (list, 'Advanced', []),
+    'IGNORED_FILES': (list, 'Advanced', []),
+    'INCLUDE_EXTRAS': (int, 'General', 0),
+    'INTERFACE': (str, 'General', 'default'),
+    'JOURNAL_MODE': (str, 'Advanced', 'wal'),
+    'KAT': (int, 'Kat', 0),
+    'KAT_PROXY_URL': (str, 'Kat', ''),
+    'KAT_RATIO': (str, 'Kat', ''),
+    'KEEP_NFO': (int, 'General', 0),
+    'KEEP_TORRENT_FILES': (int, 'General', 0),
+    'LASTFM_USERNAME': (str, 'General', ''),
+    'LAUNCH_BROWSER': (int, 'General', 1),
+    'LIBRARYSCAN': (int, 'General', 1),
+    'LIBRARYSCAN_INTERVAL': (int, 'General', 300),
+    'LMS_ENABLED': (int, 'LMS', 0),
+    'LMS_HOST': (str, 'LMS', ''),
+    'LOG_DIR': (str, 'General', ''),
+    'LOSSLESS_BITRATE_FROM': (int, 'General', 0),
+    'LOSSLESS_BITRATE_TO': (int, 'General', 0),
+    'LOSSLESS_DESTINATION_DIR': (str, 'General', ''),
+    'MB_IGNORE_AGE': (int, 'General', 365),
+    'MININOVA': (int, 'Mininova', 0),
+    'MININOVA_RATIO': (str, 'Mininova', ''),
+    'MIRROR': (str, 'General', 'musicbrainz.org'),
+    'MOVE_FILES': (int, 'General', 0),
+    'MPC_ENABLED': (bool_int, 'MPC', False),
+    'MUSIC_DIR': (str, 'General', ''),
+    'MUSIC_ENCODER': (int, 'General', 0),
+    'NEWZNAB': (int, 'Newznab', 0),
+    'NEWZNAB_APIKEY': (str, 'Newznab', ''),
+    'NEWZNAB_ENABLED': (int, 'Newznab', 1),
+    'NEWZNAB_HOST': (str, 'Newznab', ''),
+    'NMA_APIKEY': (str, 'NMA', ''),
+    'NMA_ENABLED': (int, 'NMA', 0),
+    'NMA_ONSNATCH': (int, 'NMA', 0),
+    'NMA_PRIORITY': (int, 'NMA', 0),
+    'NUMBEROFSEEDERS': (str, 'General', '10'),
+    'NZBGET_CATEGORY': (str, 'NZBget', ''),
+    'NZBGET_HOST': (str, 'NZBget', ''),
+    'NZBGET_PASSWORD': (str, 'NZBget', ''),
+    'NZBGET_PRIORITY': (int, 'NZBget', 0),
+    'NZBGET_USERNAME': (str, 'NZBget', 'nzbget'),
+    'NZBSORG': (int, 'NZBsorg', 0),
+    'NZBSORG_HASH': (str, 'NZBsorg', ''),
+    'NZBSORG_UID': (str, 'NZBsorg', ''),
+    'NZB_DOWNLOADER': (int, 'General', 0),
+    'OMGWTFNZBS': (int, 'omgwtfnzbs', 0),
+    'OMGWTFNZBS_APIKEY': (str, 'omgwtfnzbs', ''),
+    'OMGWTFNZBS_UID': (str, 'omgwtfnzbs', ''),
+    'OPEN_MAGNET_LINKS': (int, 'General', 0),  # 0: Ignore, 1: Open, 2: Convert
+    'MAGNET_LINKS': (int, 'General', 0),
+    'OSX_NOTIFY_APP': (str, 'OSX_Notify', '/Applications/Headphones'),
+    'OSX_NOTIFY_ENABLED': (int, 'OSX_Notify', 0),
+    'OSX_NOTIFY_ONSNATCH': (int, 'OSX_Notify', 0),
+    'PIRATEBAY': (int, 'Piratebay', 0),
+    'PIRATEBAY_PROXY_URL': (str, 'Piratebay', ''),
+    'PIRATEBAY_RATIO': (str, 'Piratebay', ''),
+    'OLDPIRATEBAY': (int, 'Old Piratebay', 0),
+    'OLDPIRATEBAY_URL': (str, 'Old Piratebay', ''),
+    'OLDPIRATEBAY_RATIO': (str, 'Old Piratebay', ''),
+    'PLEX_CLIENT_HOST': (str, 'Plex', ''),
+    'PLEX_ENABLED': (int, 'Plex', 0),
+    'PLEX_NOTIFY': (int, 'Plex', 0),
+    'PLEX_PASSWORD': (str, 'Plex', ''),
+    'PLEX_SERVER_HOST': (str, 'Plex', ''),
+    'PLEX_UPDATE': (int, 'Plex', 0),
+    'PLEX_USERNAME': (str, 'Plex', ''),
+    'PREFERRED_BITRATE': (str, 'General', ''),
+    'PREFERRED_BITRATE_ALLOW_LOSSLESS': (int, 'General', 0),
+    'PREFERRED_BITRATE_HIGH_BUFFER': (int, 'General', 0),
+    'PREFERRED_BITRATE_LOW_BUFFER': (int, 'General', 0),
+    'PREFERRED_QUALITY': (int, 'General', 0),
+    'PREFERRED_WORDS': (str, 'General', ''),
+    'PREFER_TORRENTS': (int, 'General', 0),
+    'PROWL_ENABLED': (int, 'Prowl', 0),
+    'PROWL_KEYS': (str, 'Prowl', ''),
+    'PROWL_ONSNATCH': (int, 'Prowl', 0),
+    'PROWL_PRIORITY': (int, 'Prowl', 0),
+    'PUSHALOT_APIKEY': (str, 'Pushalot', ''),
+    'PUSHALOT_ENABLED': (int, 'Pushalot', 0),
+    'PUSHALOT_ONSNATCH': (int, 'Pushalot', 0),
+    'PUSHBULLET_APIKEY': (str, 'PushBullet', ''),
+    'PUSHBULLET_DEVICEID': (str, 'PushBullet', ''),
+    'PUSHBULLET_ENABLED': (int, 'PushBullet', 0),
+    'PUSHBULLET_ONSNATCH': (int, 'PushBullet', 0),
+    'PUSHOVER_APITOKEN': (str, 'Pushover', ''),
+    'PUSHOVER_ENABLED': (int, 'Pushover', 0),
+    'PUSHOVER_KEYS': (str, 'Pushover', ''),
+    'PUSHOVER_ONSNATCH': (int, 'Pushover', 0),
+    'PUSHOVER_PRIORITY': (int, 'Pushover', 0),
+    'RENAME_FILES': (int, 'General', 0),
+    'REPLACE_EXISTING_FOLDERS': (int, 'General', 0),
+    'REQUIRED_WORDS': (str, 'General', ''),
+    'RUTRACKER': (int, 'Rutracker', 0),
+    'RUTRACKER_PASSWORD': (str, 'Rutracker', ''),
+    'RUTRACKER_RATIO': (str, 'Rutracker', ''),
+    'RUTRACKER_USER': (str, 'Rutracker', ''),
+    'SAB_APIKEY': (str, 'SABnzbd', ''),
+    'SAB_CATEGORY': (str, 'SABnzbd', ''),
+    'SAB_HOST': (str, 'SABnzbd', ''),
+    'SAB_PASSWORD': (str, 'SABnzbd', ''),
+    'SAB_USERNAME': (str, 'SABnzbd', ''),
+    'SAMPLINGFREQUENCY': (int, 'General', 44100),
+    'SEARCH_INTERVAL': (int, 'General', 1440),
+    'SONGKICK_APIKEY': (str, 'Songkick', 'nd1We7dFW2RqxPw8'),
+    'SONGKICK_ENABLED': (int, 'Songkick', 1),
+    'SONGKICK_FILTER_ENABLED': (int, 'Songkick', 0),
+    'SONGKICK_LOCATION': (str, 'Songkick', ''),
+    'SUBSONIC_ENABLED': (int, 'Subsonic', 0),
+    'SUBSONIC_HOST': (str, 'Subsonic', ''),
+    'SUBSONIC_PASSWORD': (str, 'Subsonic', ''),
+    'SUBSONIC_USERNAME': (str, 'Subsonic', ''),
+    'SYNOINDEX_ENABLED': (int, 'Synoindex', 0),
+    'TORRENTBLACKHOLE_DIR': (str, 'General', ''),
+    'TORRENT_DOWNLOADER': (int, 'General', 0),
+    'TORRENT_REMOVAL_INTERVAL': (int, 'General', 720),
+    'TRANSMISSION_HOST': (str, 'Transmission', ''),
+    'TRANSMISSION_PASSWORD': (str, 'Transmission', ''),
+    'TRANSMISSION_USERNAME': (str, 'Transmission', ''),
+    'TWITTER_ENABLED': (int, 'Twitter', 0),
+    'TWITTER_ONSNATCH': (int, 'Twitter', 0),
+    'TWITTER_PASSWORD': (str, 'Twitter', ''),
+    'TWITTER_PREFIX': (str, 'Twitter', 'Headphones'),
+    'TWITTER_USERNAME': (str, 'Twitter', ''),
+    'UPDATE_DB_INTERVAL': (int, 'General', 24),
+    'USENET_RETENTION': (int, 'General', '1500'),
+    'UTORRENT_HOST': (str, 'uTorrent', ''),
+    'UTORRENT_LABEL': (str, 'uTorrent', ''),
+    'UTORRENT_PASSWORD': (str, 'uTorrent', ''),
+    'UTORRENT_USERNAME': (str, 'uTorrent', ''),
+    'VERIFY_SSL_CERT': (bool_int, 'Advanced', 1),
+    'WAFFLES': (int, 'Waffles', 0),
+    'WAFFLES_PASSKEY': (str, 'Waffles', ''),
+    'WAFFLES_RATIO': (str, 'Waffles', ''),
+    'WAFFLES_UID': (str, 'Waffles', ''),
+    'WHATCD': (int, 'What.cd', 0),
+    'WHATCD_PASSWORD': (str, 'What.cd', ''),
+    'WHATCD_RATIO': (str, 'What.cd', ''),
+    'WHATCD_USERNAME': (str, 'What.cd', ''),
+    'XBMC_ENABLED': (int, 'XBMC', 0),
+    'XBMC_HOST': (str, 'XBMC', ''),
+    'XBMC_NOTIFY': (int, 'XBMC', 0),
+    'XBMC_PASSWORD': (str, 'XBMC', ''),
+    'XBMC_UPDATE': (int, 'XBMC', 0),
+    'XBMC_USERNAME': (str, 'XBMC', ''),
+    'XLDPROFILE': (str, 'General', '')
+}
+
+# pylint:disable=R0902
+# it might be nice to refactor for fewer instance variables
+class Config(object):
+    """ Wraps access to particular values in a config file """
+
+    def __init__(self, config_file):
+        """ Initialize the config with values from a file """
+        self._config_file = config_file
+        self._config = ConfigObj(self._config_file, encoding='utf-8')
+        for key in _CONFIG_DEFINITIONS.keys():
+            self.check_setting(key)
+        self.ENCODER_MULTICORE_COUNT = max(0, self.ENCODER_MULTICORE_COUNT)
+        self._upgrade()
+
+    def _define(self, name):
+        key = name.upper()
+        ini_key = name.lower()
+        definition = _CONFIG_DEFINITIONS[key]
+        if len(definition) == 3:
+            definition_type, section, default = definition
+        else:
+            definition_type, section, _, default = definition
+        return key, definition_type, section, ini_key, default
+
+    def check_section(self, section):
+        """ Check if INI section exists, if not create it """
+        if section not in self._config:
+            self._config[section] = {}
+            return True
+        else:
+            return False
+
+    def check_setting(self, key):
+        """ Cast any value in the config to the right type or use the default """
+        key, definition_type, section, ini_key, default = self._define(key)
+        self.check_section(section)
+        try:
+            my_val = definition_type(self._config[section][ini_key])
+        except Exception:
+            my_val = definition_type(default)
+            self._config[section][ini_key] = my_val
+        return my_val
+
+    def write(self):
+        """ Make a copy of the stored config and write it to the configured file """
+        new_config = ConfigObj(encoding="UTF-8")
+        new_config.filename = self._config_file
+
+        # first copy over everything from the old config, even if it is not
+        # correctly defined to keep from losing data
+        for key, subkeys in self._config.items():
+            if key not in new_config:
+                new_config[key] = {}
+            for subkey, value in subkeys.items():
+                new_config[key][subkey] = value
+
+        # next make sure that everything we expect to have defined is so
+        for key in _CONFIG_DEFINITIONS.keys():
+            key, definition_type, section, ini_key, default = self._define(key)
+            self.check_setting(key)
+            if section not in new_config:
+                new_config[section] = {}
+            new_config[section][ini_key] = self._config[section][ini_key]
+
+        # Write it to file
+        headphones.logger.info("Writing configuration to file")
+
+        try:
+            new_config.write()
+        except IOError as e:
+            headphones.logger.error("Error writing configuration file: %s", e)
+
+    def get_extra_newznabs(self):
+        """ Return the extra newznab tuples """
+        extra_newznabs = list(
+            itertools.izip(*[itertools.islice(self.EXTRA_NEWZNABS, i, None, 3)
+            for i in range(3)])
+        )
+        return extra_newznabs
+
+    def clear_extra_newznabs(self):
+        """ Forget about the configured extra newznabs """
+        self.EXTRA_NEWZNABS = []
+
+    def add_extra_newznab(self, newznab):
+        """ Add a new extra newznab """
+        extra_newznabs = self.EXTRA_NEWZNABS
+        for item in newznab:
+            extra_newznabs.append(item)
+        self.EXTRA_NEWZNABS = extra_newznabs
+
+    def __getattr__(self, name):
+        """
+        Returns something from the ini unless it is a real property
+        of the configuration object or is not all caps.
+        """
+        if not re.match(r'[A-Z_]+$', name):
+            return super(Config, self).__getattr__(name)
+        else:
+            return self.check_setting(name)
+
+    def __setattr__(self, name, value):
+        """
+        Maps all-caps properties to ini values unless they exist on the
+        configuration object.
+        """
+        if not re.match(r'[A-Z_]+$', name):
+            super(Config, self).__setattr__(name, value)
+            return value
+        else:
+            key, definition_type, section, ini_key, default = self._define(name)
+            self._config[section][ini_key] = definition_type(value)
+            return self._config[section][ini_key]
+
+    def process_kwargs(self, kwargs):
+        """
+        Given a big bunch of key value pairs, apply them to the ini.
+        """
+        for name, value in kwargs.items():
+            key, definition_type, section, ini_key, default = self._define(name)
+            self._config[section][ini_key] = definition_type(value)
+
+    def _upgrade(self):
+        """ Update folder formats in the config & bump up config version """
+        if self.CONFIG_VERSION == '0':
+            from headphones.helpers import replace_all
+            file_values = {
+                'tracknumber': 'Track',
+                'title': 'Title',
+                'artist': 'Artist',
+                'album': 'Album',
+                'year': 'Year'
+            }
+            folder_values = {
+                'artist': 'Artist',
+                'album': 'Album',
+                'year': 'Year',
+                'releasetype': 'Type',
+                'first': 'First',
+                'lowerfirst': 'first'
+            }
+            self.FILE_FORMAT = replace_all(self.FILE_FORMAT, file_values)
+            self.FOLDER_FORMAT = replace_all(self.FOLDER_FORMAT, folder_values)
+
+            self.CONFIG_VERSION = '1'
+
+        if self.CONFIG_VERSION == '1':
+            from headphones.helpers import replace_all
+            file_values = {
+                'Track': '$Track',
+                'Title': '$Title',
+                'Artist': '$Artist',
+                'Album': '$Album',
+                'Year': '$Year',
+                'track': '$track',
+                'title': '$title',
+                'artist': '$artist',
+                'album': '$album',
+                'year': '$year'
+            }
+            folder_values = {
+                'Artist': '$Artist',
+                'Album': '$Album',
+                'Year': '$Year',
+                'Type': '$Type',
+                'First': '$First',
+                'artist': '$artist',
+                'album': '$album',
+                'year': '$year',
+                'type': '$type',
+                'first': '$first'
+            }
+            self.FILE_FORMAT = replace_all(self.FILE_FORMAT, file_values)
+            self.FOLDER_FORMAT = replace_all(self.FOLDER_FORMAT, folder_values)
+            self.CONFIG_VERSION = '2'
+
+        if self.CONFIG_VERSION == '2':
+            # Update the config to use direct path to the encoder rather than the encoder folder
+            if self.ENCODERFOLDER:
+                self.ENCODER_PATH = os.path.join(self.ENCODERFOLDER, self.ENCODER)
+            self.CONFIG_VERSION = '3'
+
+        if self.CONFIG_VERSION == '3':
+            # Update the BLACKHOLE option to the NZB_DOWNLOADER format
+            if self.BLACKHOLE:
+                self.NZB_DOWNLOADER = 2
+            self.CONFIG_VERSION = '4'
+
+        # Enable Headphones Indexer if they have a VIP account
+        if self.CONFIG_VERSION == '4':
+            if self.HPUSER and self.HPPASS:
+                self.HEADPHONES_INDEXER = True
+            self.CONFIG_VERSION = '5'
+
+        if self.CONFIG_VERSION == '5':
+            if self.OPEN_MAGNET_LINKS:
+                self.MAGNET_LINKS = 2
+            self.CONFIG_VERSION = '5'

headphones/cuesplit.py

@@ -0,0 +1,669 @@
+#  This file is part of Headphones.
+#
+#  Headphones is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  Headphones is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
+
+# Most of this lifted from here: https://github.com/SzieberthAdam/gneposis-cdgrab
+
+import os
+import sys
+import re
+import subprocess
+import copy
+import glob
+
+import headphones
+from headphones import logger
+from mutagen.flac import FLAC
+
+CUE_HEADER = {
+    'genre': '^REM GENRE (.+?)$',
+    'date': '^REM DATE (.+?)$',
+    'discid': '^REM DISCID (.+?)$',
+    'comment': '^REM COMMENT (.+?)$',
+    'catalog': '^CATALOG (.+?)$',
+    'artist': '^PERFORMER (.+?)$',
+    'title': '^TITLE (.+?)$',
+    'file': '^FILE (.+?) (WAVE|FLAC)$',
+    'accurateripid': '^REM ACCURATERIPID (.+?)$'
+}
+
+CUE_TRACK = 'TRACK (\d\d) AUDIO$'
+
+CUE_TRACK_INFO = {
+    'artist': 'PERFORMER (.+?)$',
+    'title': 'TITLE (.+?)$',
+    'isrc': 'ISRC (.+?)$',
+    'index': 'INDEX (\d\d) (.+?)$'
+}
+
+ALBUM_META_FILE_NAME = 'album.dat'
+SPLIT_FILE_NAME = 'split.dat'
+
+ALBUM_META_ALBUM_BY_CUE = ('artist', 'title', 'date', 'genre')
+
+HTOA_LENGTH_TRIGGER = 3
+
+WAVE_FILE_TYPE_BY_EXTENSION = {
+    '.wav': 'Waveform Audio',
+    '.wv': 'WavPack',
+    '.ape': "Monkey's Audio",
+    '.m4a': 'Apple Lossless',
+    '.flac': 'Free Lossless Audio Codec'
+}
+
+# TODO: Only alow flac for now
+#SHNTOOL_COMPATIBLE = ('Waveform Audio', 'WavPack', 'Free Lossless Audio Codec')
+SHNTOOL_COMPATIBLE = ('Free Lossless Audio Codec')
+
+# TODO: Make this better!
+# this module-level variable is bad. :(
+CUE_META = None
+
+
+def check_splitter(command):
+    '''Check xld or shntool installed'''
+    try:
+        env = os.environ.copy()
+        if 'xld' in command:
+            env['PATH'] += os.pathsep + '/Applications'
+        elif headphones.CONFIG.CUE_SPLIT_FLAC_PATH:
+            command = os.path.join(headphones.CONFIG.CUE_SPLIT_SHNTOOL_PATH, 'shntool')
+        devnull = open(os.devnull)
+        subprocess.Popen([command], stdout=devnull, stderr=devnull, env=env).communicate()
+    except OSError as e:
+        if e.errno == os.errno.ENOENT:
+            return False
+    return True
+
+
+def split_baby(split_file, split_cmd):
+    '''Let's split baby'''
+    logger.info('Splitting %s...', split_file.decode(headphones.SYS_ENCODING, 'replace'))
+    logger.debug(subprocess.list2cmdline(split_cmd))
+
+    # Prevent Windows from opening a terminal window
+    startupinfo = None
+
+    if headphones.SYS_PLATFORM == "win32":
+        startupinfo = subprocess.STARTUPINFO()
+        try:
+            startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
+        except AttributeError:
+            startupinfo.dwFlags |= subprocess._subprocess.STARTF_USESHOWWINDOW
+
+    env = os.environ.copy()
+    if 'xld' in split_cmd:
+        env['PATH'] += os.pathsep + '/Applications'
+    elif headphones.CONFIG.CUE_SPLIT_FLAC_PATH:
+        env['PATH'] += os.pathsep + headphones.CONFIG.CUE_SPLIT_FLAC_PATH
+
+    process = subprocess.Popen(split_cmd, startupinfo=startupinfo,
+
+    stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
+    stderr=subprocess.PIPE, env=env)
+    stdout, stderr = process.communicate()
+    if process.returncode:
+        logger.error('Split failed for %s', split_file.decode(headphones.SYS_ENCODING, 'replace'))
+        out = stdout if stdout else stderr
+        logger.error('Error details: %s', out.decode(headphones.SYS_ENCODING, 'replace'))
+        return False
+    else:
+        logger.info('Split success %s', split_file.decode(headphones.SYS_ENCODING, 'replace'))
+        return True
+
+
+def check_list(list, ignore=0):
+    '''Checks a list for None elements. If list have None (after ignore index) then it should pass only if all elements
+    are None threreafter. Returns a tuple without the None entries.'''
+
+    if ignore:
+        try:
+            list[int(ignore)]
+        except:
+            raise ValueError('non-integer ignore index or ignore index not in list')
+
+    list1 = list[:ignore]
+    list2 = list[ignore:]
+
+    try:
+        first_none = list2.index(None)
+    except:
+        return tuple(list1 + list2)
+
+    for i in range(first_none, len(list2)):
+        if list2[i]:
+            raise ValueError('non-None entry after None entry in list at index {0}'.format(i))
+
+    while True:
+        list2.remove(None)
+        try:
+            list2.index(None)
+        except:
+            break
+
+    return tuple(list1 + list2)
+
+
+def trim_cue_entry(string):
+    '''Removes leading and trailing "s.'''
+    if string[0] == '"' and string[-1] == '"':
+        string = string[1:-1]
+    return string
+
+
+def int_to_str(value, length=2):
+    '''Converts integer to string eg 3 to "03"'''
+    try:
+        int(value)
+    except:
+        raise ValueError('expected an integer value')
+
+    content = str(value)
+    while len(content) < length:
+        content = '0' + content
+    return content
+
+
+class Directory:
+    def __init__(self, path):
+        self.path = path
+        self.name = os.path.split(self.path)[-1]
+        self.content = []
+        self.update()
+
+    def filter(self, classname):
+        content = []
+        for c in self.content:
+            if c.__class__.__name__ == classname:
+                content.append(c)
+        return content
+
+    def tracks(self, ext=None, split=False):
+        content = []
+        for c in self.content:
+            ext_match = False
+            if c.__class__.__name__ == 'WaveFile':
+                if not ext or (ext and ext == c.name_ext):
+                    ext_match = True
+                if ext_match and c.track_nr:
+                    if not split or (split and c.split_file):
+                        content.append(c)
+        return content
+
+    def update(self):
+        def check_match(filename):
+            for i in self.content:
+                if i.name == filename:
+                    return True
+            return False
+
+        def identify_track_number(filename):
+            if 'split-track' in filename:
+                search = re.search('split-track(\d\d)', filename)
+                if search:
+                    n = int(search.group(1))
+                    if n:
+                        return n
+            for n in range(0, 100):
+                search = re.search(int_to_str(n), filename)
+                if search:
+                    # TODO: not part of other value such as year
+                    return n
+
+        list_dir = glob.glob1(self.path, '*')
+
+        # TODO: for some reason removes only one file
+        rem_list = []
+        for i in self.content:
+            if i.name not in list_dir:
+                rem_list.append(i)
+        for i in rem_list:
+            self.content.remove(i)
+
+        for i in list_dir:
+            if not check_match(i):
+                # music file
+                if os.path.splitext(i)[-1] in WAVE_FILE_TYPE_BY_EXTENSION.keys():
+                    track_nr = identify_track_number(i)
+                    if track_nr:
+                        self.content.append(WaveFile(self.path + os.sep + i, track_nr=track_nr))
+                    else:
+                        self.content.append(WaveFile(self.path + os.sep + i))
+
+                # cue file
+                elif os.path.splitext(i)[-1] == '.cue':
+                    self.content.append(CueFile(self.path + os.sep + i))
+
+                # meta file
+                elif i == ALBUM_META_FILE_NAME:
+                    self.content.append(MetaFile(self.path + os.sep + i))
+
+                # directory
+                elif os.path.isdir(i):
+                    self.content.append(Directory(self.path + os.sep + i))
+
+                else:
+                    self.content.append(File(self.path + os.sep + i))
+
+
+class File(object):
+    def __init__(self, path):
+        self.path = path
+        self.name = os.path.split(self.path)[-1]
+
+        self.name_name = ''.join(os.path.splitext(self.name)[:-1])
+        self.name_ext = os.path.splitext(self.name)[-1]
+        self.split_file = True if self.name_name[:11] == 'split-track' else False
+
+    def get_name(self, ext=True, cmd=False):
+
+        if ext is True:
+            content = self.name
+        elif ext is False:
+            content = self.name_name
+        elif ext[0] == '.':
+            content = self.name_name + ext
+        else:
+            raise ValueError('ext parameter error')
+
+        if cmd:
+            content = content.replace(' ', '\ ')
+
+        return content
+
+
+class CueFile(File):
+    def __init__(self, path):
+
+        def header_parser():
+            global line_content
+            c = self.content.splitlines()
+            header_dict = {}
+            #remaining_headers = CUE_HEADER
+            remaining_headers = copy.copy(CUE_HEADER)
+            line_index = 0
+            match = True
+            while match:
+                match = False
+                saved_match = None
+                line_content = c[line_index]
+                for e in remaining_headers:
+                    search_result = re.search(remaining_headers[e], line_content, re.I)
+                    if search_result:
+                        search_content = trim_cue_entry(search_result.group(1))
+                        header_dict[e] = search_content
+                        saved_match = e
+                        match = True
+                        line_index += 1
+                if saved_match:
+                    del remaining_headers[saved_match]
+            return header_dict, line_index
+
+        def track_parser(start_line):
+            c = self.content.splitlines()
+            line_index = start_line
+            line_content = c[line_index]
+            search_result = re.search(CUE_TRACK, line_content, re.I)
+            if not search_result:
+                raise ValueError('inconsistent CUE sheet, TRACK expected at line {0}'.format(line_index + 1))
+            track_nr = int(search_result.group(1))
+            line_index += 1
+            next_track = False
+            track_meta = {}
+            # we make room for future indexes
+            track_meta['index'] = [None for m in range(100)]
+
+            while not next_track:
+                if line_index < len(c):
+                    line_content = c[line_index]
+
+                    artist_search = re.search(CUE_TRACK_INFO['artist'], line_content, re.I)
+                    title_search = re.search(CUE_TRACK_INFO['title'], line_content, re.I)
+                    isrc_search = re.search(CUE_TRACK_INFO['isrc'], line_content, re.I)
+                    index_search = re.search(CUE_TRACK_INFO['index'], line_content, re.I)
+
+                    if artist_search:
+                        if trim_cue_entry(artist_search.group(1)) != self.header['artist']:
+                            track_meta['artist'] = trim_cue_entry(artist_search.group(1))
+                        line_index += 1
+                    elif title_search:
+                        track_meta['title'] = trim_cue_entry(title_search.group(1))
+                        line_index += 1
+                    elif isrc_search:
+                        track_meta['isrc'] = trim_cue_entry(isrc_search.group(1))
+                        line_index += 1
+                    elif index_search:
+                        track_meta['index'][int(index_search.group(1))] = index_search.group(2)
+                        line_index += 1
+                    elif re.search(CUE_TRACK, line_content, re.I):
+                        next_track = True
+                    elif line_index == len(c) - 1 and not line_content:
+                        # last line is empty
+                        line_index += 1
+                    elif re.search('FLAGS DCP$', line_content, re.I):
+                        track_meta['dcpflag'] = True
+                        line_index += 1
+                    else:
+                        raise ValueError('unknown entry in track error, line {0}'.format(line_index + 1))
+                else:
+                    next_track = True
+
+            track_meta['index'] = check_list(track_meta['index'], ignore=1)
+
+            return track_nr, track_meta, line_index
+
+        super(CueFile, self).__init__(path)
+
+        try:
+            with open(self.name) as cue_file:
+                self.content = cue_file.read()
+        except:
+            self.content = None
+
+        if not self.content:
+            try:
+                 with open(self.name, encoding="cp1252") as cue_file:
+                     self.content = cue_file.read()
+            except:
+                raise ValueError('Cant encode CUE Sheet.')
+
+        if self.content[0] == u'\ufeff':
+            self.content = self.content[1:]
+
+        header = header_parser()
+
+        self.header = header[0]
+
+        line_index = header[1]
+
+        # we make room for tracks
+        tracks = [None for m in range(100)]
+
+        while line_index < len(self.content.splitlines()):
+            parsed_track = track_parser(line_index)
+            line_index = parsed_track[2]
+            tracks[parsed_track[0]] = parsed_track[1]
+
+        self.tracks = check_list(tracks, ignore=1)
+
+    def get_meta(self):
+        content = ''
+        for i in ALBUM_META_ALBUM_BY_CUE:
+            if self.header.get(i):
+                content += i + '\t' + self.header[i] + '\n'
+            else:
+                content += i + '\t' + '\n'
+
+        for i in range(len(self.tracks)):
+            if self.tracks[i]:
+                if self.tracks[i].get('artist'):
+                    content += 'track' + int_to_str(i) + 'artist' + '\t' + self.tracks[i].get('artist') + '\n'
+                if self.tracks[i].get('title'):
+                    content += 'track' + int_to_str(i) + 'title' + '\t' + self.tracks[i].get('title') + '\n'
+        return content
+
+    def htoa(self):
+        '''Returns true if Hidden Track exists.'''
+        if int(self.tracks[1]['index'][1][-5:-3]) >= HTOA_LENGTH_TRIGGER:
+            return True
+        return False
+
+    def breakpoints(self):
+        '''Returns track break points. Identical as CUETools' cuebreakpoints, with the exception of my standards for HTOA.'''
+        content = ''
+        for t in range(len(self.tracks)):
+            if t == 1 and not self.htoa():
+                content += ''
+            elif t >= 1:
+                t_index = self.tracks[t]['index']
+                content += t_index[1]
+                if t < (len(self.tracks) - 1):
+                    content += '\n'
+        return content
+
+
+class MetaFile(File):
+    def __init__(self, path):
+        super(MetaFile, self).__init__(path)
+        with open(self.path) as meta_file:
+            self.rawcontent = meta_file.read()
+
+        content = {}
+        content['tracks'] = [None for m in range(100)]
+
+        for l in self.rawcontent.splitlines():
+            parsed_line = re.search('^(.+?)\t(.+?)$', l)
+            if parsed_line:
+                if parsed_line.group(1)[:5] == 'track':
+                    parsed_track = re.search('^track(\d\d)(.+?)$', parsed_line.group(1))
+                    if not parsed_track:
+                        raise ValueError('Syntax error in album meta file')
+                    if not content['tracks'][int(parsed_track.group(1))]:
+                        content['tracks'][int(parsed_track.group(1))] = dict()
+                    content['tracks'][int(parsed_track.group(1))][parsed_track.group(2)] = parsed_line.group(2)
+                else:
+                    content[parsed_line.group(1)] = parsed_line.group(2)
+
+        content['tracks'] = check_list(content['tracks'], ignore=1)
+
+        self.content = content
+
+    def flac_tags(self, track_nr):
+        common_tags = dict()
+        freeform_tags = dict()
+
+        # common flac tags
+        common_tags['artist'] = self.content['artist']
+        common_tags['album'] = self.content['title']
+        common_tags['title'] = self.content['tracks'][track_nr]['title']
+        common_tags['tracknumber'] = str(track_nr)
+        common_tags['tracktotal'] = str(len(self.content['tracks']) - 1)
+        if 'date' in self.content:
+            common_tags['date'] = self.content['date']
+        if 'genre' in CUE_META.content:
+            common_tags['genre'] = CUE_META.content['genre']
+
+        #freeform tags
+        #freeform_tags['country'] = self.content['country']
+        #freeform_tags['releasedate'] = self.content['releasedate']
+
+        return common_tags, freeform_tags
+
+    def folders(self):
+        artist = self.content['artist']
+        album = self.content['date'] + ' - ' + self.content['title'] + ' (' + self.content['label'] + ' - ' + self.content['catalog'] + ')'
+        return artist, album
+
+    def complete(self):
+        '''Check MetaFile for containing all data'''
+        self.__init__(self.path)
+        for l in self.rawcontent.splitlines():
+            if re.search('^[0-9A-Za-z]+?\t$', l):
+                return False
+        return True
+
+    def count_tracks(self):
+        '''Returns tracks count'''
+        return len(self.content['tracks']) - self.content['tracks'].count(None)
+
+
+class WaveFile(File):
+    def __init__(self, path, track_nr=None):
+        super(WaveFile, self).__init__(path)
+
+        self.track_nr = track_nr
+        self.type = WAVE_FILE_TYPE_BY_EXTENSION[self.name_ext]
+
+    def filename(self, ext=None, cmd=False):
+        title = CUE_META.content['tracks'][self.track_nr]['title']
+
+        if ext:
+            if ext[0] != '.':
+                ext = '.' + ext
+        else:
+            ext = self.name_ext
+
+        f_name = int_to_str(self.track_nr) + ' - ' + title + ext
+
+        if cmd:
+            f_name = f_name.replace(' ', '\ ')
+
+        f_name = f_name.replace('!', '')
+        f_name = f_name.replace('?', '')
+        f_name = f_name.replace('/', ';')
+
+        return f_name
+
+    def tag(self):
+        if self.type == 'Free Lossless Audio Codec':
+            f = FLAC(self.name)
+            tags = CUE_META.flac_tags(self.track_nr)
+            for t in tags[0]:
+                f[t] = tags[0][t]
+            f.save()
+
+    def mutagen(self):
+        if self.type == 'Free Lossless Audio Codec':
+            return FLAC(self.name)
+
+def split(albumpath):
+    global CUE_META
+    os.chdir(albumpath)
+    base_dir = Directory(os.getcwd())
+
+    cue = None
+    wave = None
+
+    # determining correct cue file
+    # if perfect match found
+    for _cue in base_dir.filter('CueFile'):
+        for _wave in base_dir.filter('WaveFile'):
+            if _cue.header['file'] == _wave.name:
+                logger.info('CUE Sheet found: %s', _cue.name)
+                logger.info('Music file found: %s', _wave.name)
+                cue = _cue
+                wave = _wave
+    # if no perfect match found then try without extensions
+    if not cue and not wave:
+        logger.info('No match for music files, trying to match without extensions...')
+        for _cue in base_dir.filter('CueFile'):
+            for _wave in base_dir.filter('WaveFile'):
+                if ''.join(os.path.splitext(_cue.header['file'])[:-1]) == _wave.name_name:
+                    logger.info('Possible CUE Sheet found: %s', _cue.name)
+                    logger.info('CUE Sheet refers music file: %s', _cue.header['file'])
+                    logger.info('Possible Music file found: %s', _wave.name)
+                    cue = _cue
+                    wave = _wave
+                    cue.header['file'] = wave.name
+    # if still no match then raise an exception
+    if not cue and not wave:
+        raise ValueError('No music file match found!')
+
+    # Split with xld or shntool
+    splitter = 'shntool'
+    xldprofile = None
+
+    # use xld profile to split cue
+    if headphones.CONFIG.ENCODER == 'xld' and headphones.CONFIG.MUSIC_ENCODER and headphones.CONFIG.XLDPROFILE:
+        import getXldProfile
+        xldprofile, xldformat, _ = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
+        if not xldformat:
+            raise ValueError('Details for xld profile "%s" not found, cannot split cue' % (xldprofile))
+        else:
+            if headphones.CONFIG.ENCODERFOLDER:
+                splitter = os.path.join(headphones.CONFIG.ENCODERFOLDER, 'xld')
+            else:
+                splitter = 'xld'
+    # use standard xld command to split cue
+    elif sys.platform == 'darwin':
+        splitter = 'xld'
+        if not check_splitter(splitter):
+            splitter = 'shntool'
+
+    if splitter == 'shntool' and not check_splitter(splitter):
+            raise ValueError('Command not found, ensure shntool with FLAC or xld (OS X) installed')
+
+    # Determine if file can be split (only flac allowed for shntool)
+    if 'xld' in splitter and wave.name_ext not in WAVE_FILE_TYPE_BY_EXTENSION.keys() or \
+            wave.type not in SHNTOOL_COMPATIBLE:
+        raise ValueError('Cannot split, audio file has unsupported extension')
+
+    # Split with xld
+    if 'xld' in splitter:
+        cmd = [splitter]
+        cmd.extend([wave.name])
+        cmd.extend(['-c'])
+        cmd.extend([cue.name])
+        if xldprofile:
+            cmd.extend(['--profile'])
+            cmd.extend([xldprofile])
+        else:
+            cmd.extend(['-f'])
+            cmd.extend(['flac'])
+        cmd.extend(['-o'])
+        cmd.extend([base_dir.path])
+        split = split_baby(wave.name, cmd)
+    else:
+
+        # Split with shntool
+
+        # generate temporary metafile describing the cue
+        with open(ALBUM_META_FILE_NAME, mode='w') as meta_file:
+            meta_file.write(cue.get_meta())
+        base_dir.content.append(MetaFile(os.path.abspath(ALBUM_META_FILE_NAME)))
+        # check metafile for completeness
+        if not base_dir.filter('MetaFile'):
+            raise ValueError('Cue Meta file {0} missing!'.format(ALBUM_META_FILE_NAME))
+        else:
+            CUE_META = base_dir.filter('MetaFile')[0]
+
+        with open(SPLIT_FILE_NAME, mode='w') as split_file:
+            split_file.write(cue.breakpoints())
+
+        if headphones.CONFIG.CUE_SPLIT_SHNTOOL_PATH:
+            cmd = [os.path.join(headphones.CONFIG.CUE_SPLIT_SHNTOOL_PATH, 'shntool')]
+        else:
+            cmd = ['shntool']
+
+        cmd.extend(['split'])
+        cmd.extend(['-f'])
+        cmd.extend([SPLIT_FILE_NAME])
+        cmd.extend(['-o'])
+        cmd.extend(['flac'])
+        cmd.extend([wave.name])
+        split = split_baby(wave.name, cmd)
+        os.remove(SPLIT_FILE_NAME)
+        base_dir.update()
+
+        # tag FLAC files
+        if split and CUE_META.count_tracks() == len(base_dir.tracks(ext='.flac', split=True)):
+            for t in base_dir.tracks(ext='.flac', split=True):
+                logger.info('Tagging %s...', t.name)
+                t.tag()
+
+        # rename FLAC files
+        if split and CUE_META.count_tracks() == len(base_dir.tracks(ext='.flac', split=True)):
+            for t in base_dir.tracks(ext='.flac', split=True):
+                if t.name != t.filename():
+                    logger.info('Renaming %s to %s...', t.name, t.filename())
+                    os.rename(t.name, t.filename())
+
+        os.remove(ALBUM_META_FILE_NAME)
+
+    if not split:
+        raise ValueError('Failed to split, check logs')
+    else:
+        # Rename original file
+        os.rename(wave.name, wave.name + '.original')
+        return True

headphones/db.py

@@ -21,23 +21,24 @@ from __future__ import with_statement
 
 import os
 import sqlite3
-import threading
-import time
 
 import headphones
 
 from headphones import logger
 
+
 def dbFilename(filename="headphones.db"):
 
     return os.path.join(headphones.DATA_DIR, filename)
 
+
 def getCacheSize():
     #this will protect against typecasting problems produced by empty string and None settings
-    if not headphones.CACHE_SIZEMB:
+    if not headphones.CONFIG.CACHE_SIZEMB:
         #sqlite will work with this (very slowly)
         return 0
-    return int(headphones.CACHE_SIZEMB)
+    return int(headphones.CONFIG.CACHE_SIZEMB)
+
 
 class DBConnection:
 
@@ -48,25 +49,25 @@ class DBConnection:
         #don't wait for the disk to finish writing
         self.connection.execute("PRAGMA synchronous = OFF")
         #journal disabled since we never do rollbacks
-        self.connection.execute("PRAGMA journal_mode = %s" % headphones.JOURNAL_MODE)
+        self.connection.execute("PRAGMA journal_mode = %s" % headphones.CONFIG.JOURNAL_MODE)
         #64mb of cache memory,probably need to make it user configurable
-        self.connection.execute("PRAGMA cache_size=-%s" % (getCacheSize()*1024))
+        self.connection.execute("PRAGMA cache_size=-%s" % (getCacheSize() * 1024))
         self.connection.row_factory = sqlite3.Row
 
     def action(self, query, args=None):
 
-        if query == None:
+        if query is None:
             return
 
         sqlResult = None
-        
+
         try:
             with self.connection as c:
-                if args == None:
+                if args is None:
                     sqlResult = c.execute(query)
                 else:
                     sqlResult = c.execute(query, args)
-                    
+
         except sqlite3.OperationalError, e:
             if "unable to open database file" in e.message or "database is locked" in e.message:
                 logger.warn('Database Error: %s', e)
@@ -77,14 +78,14 @@ class DBConnection:
         except sqlite3.DatabaseError, e:
             logger.error('Fatal Error executing %s :: %s', query, e)
             raise
-        
+
         return sqlResult
 
     def select(self, query, args=None):
 
         sqlResults = self.action(query, args).fetchall()
-        
+
-        if sqlResults == None or sqlResults == [None]:
+        if sqlResults is None or sqlResults == [None]:
             return []
 
         return sqlResults
@@ -93,13 +94,18 @@ class DBConnection:
 
         changesBefore = self.connection.total_changes
 
-        genParams = lambda myDict : [x + " = ?" for x in myDict.keys()]
+        genParams = lambda myDict: [x + " = ?" for x in myDict.keys()]
 
-        query = "UPDATE "+tableName+" SET " + ", ".join(genParams(valueDict)) + " WHERE " + " AND ".join(genParams(keyDict))
+        update_query = "UPDATE " + tableName + " SET " + ", ".join(genParams(valueDict)) + " WHERE " + " AND ".join(genParams(keyDict))
 
-        self.action(query, valueDict.values() + keyDict.values())
+        self.action(update_query, valueDict.values() + keyDict.values())
 
         if self.connection.total_changes == changesBefore:
-            query = "INSERT INTO "+tableName+" (" + ", ".join(valueDict.keys() + keyDict.keys()) + ")" + \
+            insert_query = (
-                        " VALUES (" + ", ".join(["?"] * len(valueDict.keys() + keyDict.keys())) + ")"
+                "INSERT INTO " + tableName + " (" + ", ".join(valueDict.keys() + keyDict.keys()) + ")" +
-            self.action(query, valueDict.values() + keyDict.values())
+                " VALUES (" + ", ".join(["?"] * len(valueDict.keys() + keyDict.keys())) + ")"
+            )
+            try:
+                self.action(insert_query, valueDict.values() + keyDict.values())
+            except sqlite3.IntegrityError:
+                logger.info('Queries failed: %s and %s', update_query, insert_query)

headphones/exceptions.py

@@ -13,11 +13,13 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
+
 class HeadphonesException(Exception):
     """
     Generic Headphones Exception - should never be thrown, only subclassed
     """
 
+
 class NewzbinAPIThrottled(HeadphonesException):
     """
     Newzbin has throttled us, deal with it

headphones/getXldProfile.py

@@ -1,32 +1,31 @@
 import os.path
-import plistlib
+import biplist
-import sys
-import xml.parsers.expat as expat
-import commands
 from headphones import logger
 
+
 def getXldProfile(xldProfile):
+
     xldProfileNotFound = xldProfile
-    expandedPath = os.path.expanduser('~/Library/Preferences/jp.tmkk.XLD.plist')
+
-    try:
+    expanded = os.path.expanduser('~/Library/Preferences/jp.tmkk.XLD.plist')
-        preferences = plistlib.Plist.fromFile(expandedPath)
+    if not os.path.isfile(expanded):
-    except (expat.ExpatError):
+        logger.warn("Could not find xld preferences at: %s", expanded)
-        os.system("/usr/bin/plutil -convert xml1 %s" % expandedPath )
-        try:
-            preferences = plistlib.Plist.fromFile(expandedPath)
-        except (ImportError):
-            os.system("/usr/bin/plutil -convert binary1 %s" % expandedPath )
-            logger.info('The plist at "%s" has a date in it, and therefore is not useable.' % expandedPath)
-            return(xldProfileNotFound, None, None)
-    except (ImportError):
-        logger.info('The plist at "%s" has a date in it, and therefore is not useable.' % expandedPath)
-    except:
-        logger.info('Unexpected error:', sys.exc_info()[0])
         return(xldProfileNotFound, None, None)
 
-    xldProfile = xldProfile.lower()
+    # Get xld preferences plist
-    profiles = preferences.get('Profiles')
+    try:
+        preferences = biplist.readPlist(expanded)
+    except (biplist.InvalidPlistException, biplist.NotBinaryPlistException), e:
+        logger.error("Error reading xld preferences plist: %s", e)
+        return(xldProfileNotFound, None, None)
 
+    if not isinstance(preferences, dict):
+        logger.error("Error reading xld preferences plist, not a dict: %r", preferences)
+        return(xldProfileNotFound, None, None)
+
+    profiles = preferences.get('Profiles', []) # pylint:disable=E1103
+
+    xldProfile = xldProfile.lower()
     for profile in profiles:
 
         profilename = profile.get('XLDProfileManager_ProfileName')
@@ -178,4 +177,4 @@ def getXldProfile(xldProfile):
 
             return(xldProfileForCmd, xldFormat, xldBitrate)
 
-    return(xldProfileNotFound, None, None)
+    return(xldProfileNotFound, None, None)

headphones/helpers.py

@@ -13,17 +13,19 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
+from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
-import re
-import time
-import shutil
-import datetime
-import headphones
-import unicodedata
-import sys
 
 from operator import itemgetter
-from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
+
+import unicodedata
+import headphones
+import datetime
+import fnmatch
+import shutil
+import time
+import sys
+import re
+import os
 
 # Modified from https://github.com/Verrus/beets-plugin-featInTitle
 RE_FEATURING = re.compile(r"[fF]t\.|[fF]eaturing|[fF]eat\.|\b[wW]ith\b|&|vs\.")
@@ -31,8 +33,9 @@ RE_FEATURING = re.compile(r"[fF]t\.|[fF]eaturing|[fF]eat\.|\b[wW]ith\b|&|vs\.")
 RE_CD_ALBUM = re.compile(r"\(?((CD|disc)\s*[0-9]+)\)?", re.I)
 RE_CD = re.compile(r"^(CD|dics)\s*[0-9]+$", re.I)
 
+
 def multikeysort(items, columns):
-    comparers = [ ((itemgetter(col[1:].strip()), -1) if col.startswith('-') else (itemgetter(col.strip()), 1)) for col in columns]
+    comparers = [((itemgetter(col[1:].strip()), -1) if col.startswith('-') else (itemgetter(col.strip()), 1)) for col in columns]
 
     def comparer(left, right):
         for fn, mult in comparers:
@@ -44,12 +47,14 @@ def multikeysort(items, columns):
 
     return sorted(items, cmp=comparer)
 
+
 def checked(variable):
     if variable:
         return 'Checked'
     else:
         return ''
 
+
 def radio(variable, pos):
 
     if variable == pos:
@@ -57,40 +62,42 @@ def radio(variable, pos):
     else:
         return ''
 
+
 def latinToAscii(unicrap):
     """
     From couch potato
     """
-    xlate = {0xc0:'A', 0xc1:'A', 0xc2:'A', 0xc3:'A', 0xc4:'A', 0xc5:'A',
+    xlate = {
-        0xc6:'Ae', 0xc7:'C',
+        0xc0: 'A', 0xc1: 'A', 0xc2: 'A', 0xc3: 'A', 0xc4: 'A', 0xc5: 'A',
-        0xc8:'E', 0xc9:'E', 0xca:'E', 0xcb:'E', 0x86:'e',
+        0xc6: 'Ae', 0xc7: 'C',
-        0xcc:'I', 0xcd:'I', 0xce:'I', 0xcf:'I',
+        0xc8: 'E', 0xc9: 'E', 0xca: 'E', 0xcb: 'E', 0x86: 'e',
-        0xd0:'Th', 0xd1:'N',
+        0xcc: 'I', 0xcd: 'I', 0xce: 'I', 0xcf: 'I',
-        0xd2:'O', 0xd3:'O', 0xd4:'O', 0xd5:'O', 0xd6:'O', 0xd8:'O',
+        0xd0: 'Th', 0xd1: 'N',
-        0xd9:'U', 0xda:'U', 0xdb:'U', 0xdc:'U',
+        0xd2: 'O', 0xd3: 'O', 0xd4: 'O', 0xd5: 'O', 0xd6: 'O', 0xd8: 'O',
-        0xdd:'Y', 0xde:'th', 0xdf:'ss',
+        0xd9: 'U', 0xda: 'U', 0xdb: 'U', 0xdc: 'U',
-        0xe0:'a', 0xe1:'a', 0xe2:'a', 0xe3:'a', 0xe4:'a', 0xe5:'a',
+        0xdd: 'Y', 0xde: 'th', 0xdf: 'ss',
-        0xe6:'ae', 0xe7:'c',
+        0xe0: 'a', 0xe1: 'a', 0xe2: 'a', 0xe3: 'a', 0xe4: 'a', 0xe5: 'a',
-        0xe8:'e', 0xe9:'e', 0xea:'e', 0xeb:'e', 0x0259:'e',
+        0xe6: 'ae', 0xe7: 'c',
-        0xec:'i', 0xed:'i', 0xee:'i', 0xef:'i',
+        0xe8: 'e', 0xe9: 'e', 0xea: 'e', 0xeb: 'e', 0x0259: 'e',
-        0xf0:'th', 0xf1:'n',
+        0xec: 'i', 0xed: 'i', 0xee: 'i', 0xef: 'i',
-        0xf2:'o', 0xf3:'o', 0xf4:'o', 0xf5:'o', 0xf6:'o', 0xf8:'o',
+        0xf0: 'th', 0xf1: 'n',
-        0xf9:'u', 0xfa:'u', 0xfb:'u', 0xfc:'u',
+        0xf2: 'o', 0xf3: 'o', 0xf4: 'o', 0xf5: 'o', 0xf6: 'o', 0xf8: 'o',
-        0xfd:'y', 0xfe:'th', 0xff:'y',
+        0xf9: 'u', 0xfa: 'u', 0xfb: 'u', 0xfc: 'u',
-        0xa1:'!', 0xa2:'{cent}', 0xa3:'{pound}', 0xa4:'{currency}',
+        0xfd: 'y', 0xfe: 'th', 0xff: 'y',
-        0xa5:'{yen}', 0xa6:'|', 0xa7:'{section}', 0xa8:'{umlaut}',
+        0xa1: '!', 0xa2: '{cent}', 0xa3: '{pound}', 0xa4: '{currency}',
-        0xa9:'{C}', 0xaa:'{^a}', 0xab:'<<', 0xac:'{not}',
+        0xa5: '{yen}', 0xa6: '|', 0xa7: '{section}', 0xa8: '{umlaut}',
-        0xad:'-', 0xae:'{R}', 0xaf:'_', 0xb0:'{degrees}',
+        0xa9: '{C}', 0xaa: '{^a}', 0xab: '<<', 0xac: '{not}',
-        0xb1:'{+/-}', 0xb2:'{^2}', 0xb3:'{^3}', 0xb4:"'",
+        0xad: '-', 0xae: '{R}', 0xaf: '_', 0xb0: '{degrees}',
-        0xb5:'{micro}', 0xb6:'{paragraph}', 0xb7:'*', 0xb8:'{cedilla}',
+        0xb1: '{+/-}', 0xb2: '{^2}', 0xb3: '{^3}', 0xb4: "'",
-        0xb9:'{^1}', 0xba:'{^o}', 0xbb:'>>',
+        0xb5: '{micro}', 0xb6: '{paragraph}', 0xb7: '*', 0xb8: '{cedilla}',
-        0xbc:'{1/4}', 0xbd:'{1/2}', 0xbe:'{3/4}', 0xbf:'?',
+        0xb9: '{^1}', 0xba: '{^o}', 0xbb: '>>',
-        0xd7:'*', 0xf7:'/'
+        0xbc: '{1/4}', 0xbd: '{1/2}', 0xbe: '{3/4}', 0xbf: '?',
-        }
+        0xd7: '*', 0xf7: '/'
+    }
 
     r = ''
     for i in unicrap:
-        if xlate.has_key(ord(i)):
+        if ord(i) in xlate:
             r += xlate[ord(i)]
         elif ord(i) >= 0x80:
             pass
@@ -98,9 +105,10 @@ def latinToAscii(unicrap):
             r += str(i)
     return r
 
+
 def convert_milliseconds(ms):
 
-    seconds = ms/1000
+    seconds = ms / 1000
     gmtime = time.gmtime(seconds)
     if seconds > 3600:
         minutes = time.strftime("%H:%M:%S", gmtime)
@@ -109,6 +117,7 @@ def convert_milliseconds(ms):
 
     return minutes
 
+
 def convert_seconds(s):
 
     gmtime = time.gmtime(s)
@@ -119,15 +128,18 @@ def convert_seconds(s):
 
     return minutes
 
+
 def today():
     today = datetime.date.today()
     yyyymmdd = datetime.date.isoformat(today)
     return yyyymmdd
 
+
 def now():
     now = datetime.datetime.now()
     return now.strftime("%Y-%m-%d %H:%M:%S")
 
+
 def get_age(date):
 
     try:
@@ -136,33 +148,43 @@ def get_age(date):
         return False
 
     try:
-        days_old = int(split_date[0])*365 + int(split_date[1])*30 + int(split_date[2])
+        days_old = int(split_date[0]) * 365 + int(split_date[1]) * 30 + int(split_date[2])
     except IndexError:
         days_old = False
 
     return days_old
 
+
 def bytes_to_mb(bytes):
 
-    mb = int(bytes)/1048576
+    mb = int(bytes) / 1048576
     size = '%.1f MB' % mb
     return size
 
+
 def mb_to_bytes(mb_str):
     result = re.search('^(\d+(?:\.\d+)?)\s?(?:mb)?', mb_str, flags=re.I)
     if result:
-        return int(float(result.group(1))*1048576)
+        return int(float(result.group(1)) * 1048576)
+
 
 def piratesize(size):
     split = size.split(" ")
     factor = float(split[0])
-    unit = split[1]
+    unit = split[1].upper()
+    
     if unit == 'MiB':
         size = factor * 1048576
+    elif unit == 'MB':
+        size = factor * 1000000
     elif unit == 'GiB':
         size = factor * 1073741824
+    elif unit == 'GB':
+        size = factor * 1000000000
     elif unit == 'KiB':
         size = factor * 1024
+    elif unit == 'KB':
+        size = factor * 1000
     elif unit == "B":
         size = factor
     else:
@@ -170,6 +192,7 @@ def piratesize(size):
 
     return size
 
+
 def replace_all(text, dic, normalize=False):
 
     if not text:
@@ -177,13 +200,17 @@ def replace_all(text, dic, normalize=False):
 
     for i, j in dic.iteritems():
         if normalize:
-            if sys.platform == 'darwin':
+            try:
-                j = unicodedata.normalize('NFD', j)
+                if sys.platform == 'darwin':
-            else:
+                    j = unicodedata.normalize('NFD', j)
-                j = unicodedata.normalize('NFC', j)
+                else:
+                    j = unicodedata.normalize('NFC', j)
+            except TypeError:
+                j = unicodedata.normalize('NFC', j.decode(headphones.SYS_ENCODING, 'replace'))
         text = text.replace(i, j)
     return text
 
+
 def replace_illegal_chars(string, type="file"):
     if type == "file":
         string = re.sub('[\?"*:|<>/]', '_', string)
@@ -192,6 +219,7 @@ def replace_illegal_chars(string, type="file"):
 
     return string
 
+
 def cleanName(string):
 
     pass1 = latinToAscii(string).lower()
@@ -199,6 +227,7 @@ def cleanName(string):
 
     return out_string
 
+
 def cleanTitle(title):
 
     title = re.sub('[\.\-\/\_]', ' ', title).lower()
@@ -210,6 +239,7 @@ def cleanTitle(title):
 
     return title
 
+
 def split_path(f):
     """
     Split a path into components, starting with the drive letter (if any). Given
@@ -219,7 +249,7 @@ def split_path(f):
     components = []
     drive, path = os.path.splitdrive(f)
 
-    # Stip the folder from the path, iterate until nothing is left
+    # Strip the folder from the path, iterate until nothing is left
     while True:
         path, folder = os.path.split(path)
 
@@ -241,6 +271,7 @@ def split_path(f):
     # Done
     return components
 
+
 def expand_subfolders(f):
     """
     Try to expand a given folder and search for subfolders containing media
@@ -269,7 +300,7 @@ def expand_subfolders(f):
         return
 
     # Split into path components
-    media_folders = [ split_path(media_folder) for media_folder in media_folders ]
+    media_folders = [split_path(media_folder) for media_folder in media_folders]
 
     # Correct folder endings such as CD1 etc.
     for index, media_folder in enumerate(media_folders):
@@ -277,7 +308,7 @@ def expand_subfolders(f):
             media_folders[index] = media_folders[index][:-1]
 
     # Verify the result by computing path depth relative to root.
-    path_depths = [ len(media_folder) for media_folder in media_folders ]
+    path_depths = [len(media_folder) for media_folder in media_folders]
     difference = max(path_depths) - min(path_depths)
 
     if difference > 0:
@@ -287,15 +318,15 @@ def expand_subfolders(f):
         # directory may contain separate CD's and maybe some extra's. The
         # structure may look like X albums at same depth, and (one or more)
         # extra folders with a higher depth.
-        extra_media_folders = [ media_folder[:min(path_depths)] for media_folder in media_folders if len(media_folder) > min(path_depths) ]
+        extra_media_folders = [media_folder[:min(path_depths)] for media_folder in media_folders if len(media_folder) > min(path_depths)]
-        extra_media_folders = list(set([ os.path.join(*media_folder) for media_folder in extra_media_folders ]))
+        extra_media_folders = list(set([os.path.join(*media_folder) for media_folder in extra_media_folders]))
 
         logger.info("Please look at the following folder(s), since they cause the depth difference: %s", extra_media_folders)
         return
 
     # Convert back to paths and remove duplicates, which may be there after
     # correcting the paths
-    media_folders = list(set([ os.path.join(*media_folder) for media_folder in media_folders ]))
+    media_folders = list(set([os.path.join(*media_folder) for media_folder in media_folders]))
 
     # Don't return a result if the number of subfolders is one. In this case,
     # this algorithm will not improve processing and will likely interfere
@@ -307,23 +338,53 @@ def expand_subfolders(f):
     logger.debug("Expanded subfolders in folder: %s", media_folders)
     return media_folders
 
+
+def path_match_patterns(path, patterns):
+    """
+    Check if a path matches one or more patterns. The whole path will be
+    matched be matched against the patterns.
+    """
+
+    for pattern in patterns:
+        if fnmatch.fnmatch(path, pattern):
+            return True
+
+    # No match
+    return False
+
+
+def path_filter_patterns(paths, patterns, root=None):
+    """
+    Scan for ignored paths based on glob patterns. Note that the whole path
+    will be matched, therefore paths should only contain the relative paths.
+
+    The root is optional, and only used for producing meaningful debug info.
+    """
+
+    from headphones import logger
+
+    ignored = 0
+
+    for path in paths[:]:
+        if path_match_patterns(path, patterns):
+            logger.debug("Path ignored by pattern: %s",
+                os.path.join(root or "", path))
+
+            ignored += 1
+            paths.remove(path)
+
+    # Return number of ignored paths
+    return ignored
+
+
 def extract_data(s):
 
     s = s.replace('_', ' ')
 
     #headphones default format
-    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\[(?P<year>.*?)\]', re.VERBOSE)
+    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s[\[\(](?P<year>.*?)[\]\)]', re.VERBOSE)
     match = pattern.match(s)
 
-    if match:
-        name = match.group("name")
-        album = match.group("album")
-        year = match.group("year")
-        return (name, album, year)
-
-    #newzbin default format
-    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\((?P<year>\d+?\))', re.VERBOSE)
-    match = pattern.match(s)
     if match:
         name = match.group("name")
         album = match.group("album")
@@ -343,6 +404,7 @@ def extract_data(s):
     else:
         return (None, None, None)
 
+
 def extract_metadata(f):
     """
     Scan all files in the given directory and decide on an artist, album and
@@ -392,9 +454,9 @@ def extract_metadata(f):
         return (None, None, None)
 
     # Count distinct values
-    artists = list(set([ x[0] for x in results ]))
+    artists = list(set([x[0] for x in results]))
-    albums = list(set([ x[1] for x in results ]))
+    albums = list(set([x[1] for x in results]))
-    years = list(set([ x[2] for x in results ]))
+    years = list(set([x[2] for x in results]))
 
     # Remove things such as CD2 from album names
     if len(albums) > 1:
@@ -422,8 +484,8 @@ def extract_metadata(f):
 
     # (Lots of) different artists. Could be a featuring album, so test for this.
     if len(artists) > 1 and len(albums) == 1:
-        split_artists = [ RE_FEATURING.split(artist) for artist in artists ]
+        split_artists = [RE_FEATURING.split(x) for x in artists]
-        featurings = [ len(split_artist) - 1 for split_artist in split_artists ]
+        featurings = [len(split_artist) - 1 for split_artist in split_artists]
         logger.info("Album seem to feature %d different artists", sum(featurings))
 
         if sum(featurings) > 0:
@@ -441,6 +503,79 @@ def extract_metadata(f):
 
     return (None, None, None)
 
+
+def get_downloaded_track_list(albumpath):
+    """
+     Return a list of audio files for the given directory.
+     """
+    downloaded_track_list = []
+
+    for root, dirs, files in os.walk(albumpath):
+        for _file in files:
+            extension = os.path.splitext(_file)[1].lower()[1:]
+            if extension in headphones.MEDIA_FORMATS:
+                downloaded_track_list.append(os.path.join(root, _file))
+
+    return downloaded_track_list
+
+
+def preserve_torrent_direcory(albumpath):
+    """
+    Copy torrent directory to headphones-modified to keep files for seeding.
+    """
+    from headphones import logger
+    new_folder = os.path.join(albumpath, 'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
+    logger.info("Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")
+    try:
+        shutil.copytree(albumpath, new_folder)
+        return new_folder
+    except Exception as e:
+        logger.warn("Cannot copy/move files to temp folder: " + \
+                    new_folder.decode(headphones.SYS_ENCODING, 'replace') + \
+                    ". Not continuing. Error: " + str(e))
+        return None
+
+
+def cue_split(albumpath):
+    """
+     Attempts to check and split audio files by a cue for the given directory.
+     """
+    # Walk directory and scan all media files
+    count = 0
+    cue_count = 0
+    cue_dirs = []
+
+    for root, dirs, files in os.walk(albumpath):
+        for _file in files:
+            extension = os.path.splitext(_file)[1].lower()[1:]
+            if extension in headphones.MEDIA_FORMATS:
+                count += 1
+            elif extension == 'cue':
+                cue_count += 1
+                if root not in cue_dirs:
+                    cue_dirs.append(root)
+
+    # Split cue
+    if cue_count and cue_count >= count and cue_dirs:
+
+        from headphones import logger, cuesplit
+        logger.info("Attempting to split audio files by cue")
+
+        cwd = os.getcwd()
+        for cue_dir in cue_dirs:
+            try:
+                cuesplit.split(cue_dir)
+            except Exception as e:
+                os.chdir(cwd)
+                logger.warn("Cue not split: " + str(e))
+                return False
+
+        os.chdir(cwd)
+        return True
+
+    return False
+
+
 def extract_logline(s):
     # Default log format
     pattern = re.compile(r'(?P<timestamp>.*?)\s\-\s(?P<level>.*?)\s*\:\:\s(?P<thread>.*?)\s\:\s(?P<message>.*)', re.VERBOSE)
@@ -454,14 +589,11 @@ def extract_logline(s):
     else:
         return None
 
+
 def extract_song_data(s):
+    from headphones import logger
 
     #headphones default format
-    music_dir = headphones.MUSIC_DIR
-    folder_format = headphones.FOLDER_FORMAT
-    file_format = headphones.FILE_FORMAT
-
-    full_format = os.path.join(headphones.MUSIC_DIR)
     pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\[(?P<year>.*?)\]', re.VERBOSE)
     match = pattern.match(s)
 
@@ -485,6 +617,7 @@ def extract_song_data(s):
         logger.info("Couldn't parse %s into a valid Newbin format", s)
         return (name, album, year)
 
+
 def smartMove(src, dest, delete=True):
 
     from headphones import logger
@@ -506,7 +639,7 @@ def smartMove(src, dest, delete=True):
                 try:
                     os.rename(src, os.path.join(source_dir, newfile))
                     filename = newfile
-                except Exception, e:
+                except Exception as e:
                     logger.warn('Error renaming %s: %s', src.decode(headphones.SYS_ENCODING, 'replace'), e)
                 break
 
@@ -516,41 +649,82 @@ def smartMove(src, dest, delete=True):
         else:
             shutil.copy(os.path.join(source_dir, filename), os.path.join(dest, filename))
             return True
-    except Exception, e:
+    except Exception as e:
         logger.warn('Error moving file %s: %s', filename.decode(headphones.SYS_ENCODING, 'replace'), e)
 
+def walk_directory(basedir, followlinks=True):
+    """
+    Enhanced version of 'os.walk' where symlink directores are traversed, but
+    with care. In case a folder is already processed, don't traverse it again.
+    """
+
+    import logger
+
+    # Add the base path, because symlinks poiting to the basedir should not be
+    # traversed again.
+    traversed = [os.path.abspath(basedir)]
+
+    def _inner(root, directories, files):
+        for directory in directories:
+            path = os.path.join(root, directory)
+
+            if followlinks and os.path.islink(path):
+                real_path = os.path.abspath(os.readlink(path))
+
+                if real_path in traversed:
+                    logger.debug("Skipping '%s' since it is a symlink to "\
+                        "'%s', which is already visited.", path, real_path)
+                else:
+                    traversed.append(real_path)
+
+                    for args in os.walk(real_path):
+                        for result in _inner(*args):
+                            yield result
+
+        # Pass on actual result
+        yield root, directories, files
+
+    # Start traversing
+    for args in os.walk(basedir):
+        for result in _inner(*args):
+            yield result
+
 #########################
 #Sab renaming functions #
 #########################
 
 # TODO: Grab config values from sab to know when these options are checked. For now we'll just iterate through all combinations
 
+
 def sab_replace_dots(name):
-    return name.replace('.',' ')
+    return name.replace('.', ' ')
+
+
 def sab_replace_spaces(name):
-    return name.replace(' ','_')
+    return name.replace(' ', '_')
+
 
 def sab_sanitize_foldername(name):
     """ Return foldername with dodgy chars converted to safe ones
         Remove any leading and trailing dot and space characters
     """
     CH_ILLEGAL = r'\/<>?*|"'
-    CH_LEGAL   = r'++{}!@#`'
+    CH_LEGAL = r'++{}!@#`'
 
     FL_ILLEGAL = CH_ILLEGAL + ':\x92"'
-    FL_LEGAL   = CH_LEGAL +   "-''"
+    FL_LEGAL = CH_LEGAL + "-''"
 
     uFL_ILLEGAL = FL_ILLEGAL.decode('latin-1')
-    uFL_LEGAL   = FL_LEGAL.decode('latin-1')
+    uFL_LEGAL = FL_LEGAL.decode('latin-1')
 
     if not name:
         return name
     if isinstance(name, unicode):
         illegal = uFL_ILLEGAL
-        legal   = uFL_LEGAL
+        legal = uFL_LEGAL
     else:
         illegal = FL_ILLEGAL
-        legal   = FL_LEGAL
+        legal = FL_LEGAL
 
     lst = []
     for ch in name.strip():
@@ -579,34 +753,33 @@ def split_string(mystring, splitvar=','):
 
 def create_https_certificates(ssl_cert, ssl_key):
     """
-    Stolen from SickBeard (http://github.com/midgetspy/Sick-Beard):
+    Create a pair of self-signed HTTPS certificares and store in them in
-    Create self-signed HTTPS certificares and store in paths 'ssl_cert' and 'ssl_key'
+    'ssl_cert' and 'ssl_key'. Method assumes pyOpenSSL is installed.
+
+    This code is stolen from SickBeard (http://github.com/midgetspy/Sick-Beard).
     """
+
     from headphones import logger
 
-    try:
+    from OpenSSL import crypto
-        from OpenSSL import crypto
+    from certgen import createKeyPair, createCertRequest, createCertificate, \
-        from lib.certgen import createKeyPair, createCertRequest, createCertificate, TYPE_RSA, serial
+        TYPE_RSA, serial
-    except:
-        logger.warn("pyOpenSSL module missing, please install to enable HTTPS")
-        return False
 
     # Create the CA Certificate
-    cakey = createKeyPair(TYPE_RSA, 1024)
+    cakey = createKeyPair(TYPE_RSA, 2048)
-    careq = createCertRequest(cakey, CN='Certificate Authority')
+    careq = createCertRequest(cakey, CN="Certificate Authority")
-    cacert = createCertificate(careq, (careq, cakey), serial, (0, 60*60*24*365*10)) # ten years
+    cacert = createCertificate(careq, (careq, cakey), serial, (0, 60 * 60 * 24 * 365 * 10)) # ten years
 
-    cname = 'Headphones'
+    pkey = createKeyPair(TYPE_RSA, 2048)
-    pkey = createKeyPair(TYPE_RSA, 1024)
+    req = createCertRequest(pkey, CN="Headphones")
-    req = createCertRequest(pkey, CN=cname)
+    cert = createCertificate(req, (cacert, cakey), serial, (0, 60 * 60 * 24 * 365 * 10)) # ten years
-    cert = createCertificate(req, (cacert, cakey), serial, (0, 60*60*24*365*10)) # ten years
 
     # Save the key and certificate to disk
     try:
-        with open(ssl_key, 'w') as f:
+        with open(ssl_key, "w") as fp:
-            f.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, pkey))
+            fp.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, pkey))
-        with open(ssl_cert, 'w') as f:
+        with open(ssl_cert, "w") as fp:
-            f.write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
+            fp.write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
     except IOError as e:
         logger.error("Error creating SSL key and certificate: %s", e)
         return False

headphones/importer.py

@@ -17,13 +17,11 @@ from headphones import logger, helpers, db, mb, lastfm
 
 from beets.mediafile import MediaFile
 
-import os
 import time
-import threading
 import headphones
 
 blacklisted_special_artist_names = ['[anonymous]', '[data]', '[no artist]',
-    '[traditional]','[unknown]','Various Artists']
+    '[traditional]', '[unknown]', 'Various Artists']
 blacklisted_special_artists = ['f731ccc4-e22a-43af-a747-64213329e088',
     '33cf029c-63b0-41a0-9855-be2a3665fb3b',
     '314e1c25-dde7-4e4d-b2f4-0a7b9f7c56dc',
@@ -32,6 +30,7 @@ blacklisted_special_artists = ['f731ccc4-e22a-43af-a747-64213329e088',
     '125ec42a-7229-4250-afc5-e057484327fe',
     '89ad4ac3-39f7-470e-963a-56509c546377']
 
+
 def is_exists(artistid):
     myDB = db.DBConnection()
 
@@ -49,10 +48,9 @@ def artistlist_to_mbids(artistlist, forced=False):
 
     for artist in artistlist:
 
-        if not artist and not (artist == ' '):
+        if not artist and artist != ' ':
             continue
 
-
         # If adding artists through Manage New Artists, they're coming through as non-unicode (utf-8?)
         # and screwing everything up
         if not isinstance(artist, unicode):
@@ -105,12 +103,14 @@ def artistlist_to_mbids(artistlist, forced=False):
     except Exception as e:
         logger.warn('Failed to update arist information from Last.fm: %s' % e)
 
+
 def addArtistIDListToDB(artistidlist):
     # Used to add a list of artist IDs to the database in a single thread
     logger.debug("Importer: Adding artist ids %s" % artistidlist)
     for artistid in artistidlist:
         addArtisttoDB(artistid)
 
+
 def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
     # Putting this here to get around the circular import. We're using this to update thumbnails for artist/albums
@@ -131,19 +131,19 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
     # We need the current minimal info in the database instantly
     # so we don't throw a 500 error when we redirect to the artistPage
-    controlValueDict = {"ArtistID":     artistid}
+    controlValueDict = {"ArtistID": artistid}
 
     # Don't replace a known artist name with an "Artist ID" placeholder
     dbartist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [artistid]).fetchone()
 
     # Only modify the Include Extras stuff if it's a new artist. We need it early so we know what to fetch
     if not dbartist:
-        newValueDict = {"ArtistName":   "Artist ID: %s" % (artistid),
+        newValueDict = {"ArtistName": "Artist ID: %s" % (artistid),
-                        "Status":       "Loading",
+                        "Status": "Loading",
-                        "IncludeExtras": headphones.INCLUDE_EXTRAS,
+                        "IncludeExtras": headphones.CONFIG.INCLUDE_EXTRAS,
-                        "Extras":        headphones.EXTRAS }
+                        "Extras": headphones.CONFIG.EXTRAS}
     else:
-        newValueDict = {"Status":   "Loading"}
+        newValueDict = {"Status": "Loading"}
 
     myDB.upsert("artists", newValueDict, controlValueDict)
 
@@ -160,10 +160,10 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
     if not artist:
         logger.warn("Error fetching artist info. ID: " + artistid)
         if dbartist is None:
-            newValueDict = {"ArtistName":   "Fetch failed, try refreshing. (%s)" % (artistid),
+            newValueDict = {"ArtistName": "Fetch failed, try refreshing. (%s)" % (artistid),
-                    "Status":   "Active"}
+                    "Status": "Active"}
         else:
-            newValueDict = {"Status":   "Active"}
+            newValueDict = {"Status": "Active"}
         myDB.upsert("artists", newValueDict, controlValueDict)
         return
 
@@ -172,13 +172,12 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
     else:
         sortname = artist['artist_name']
 
-
     logger.info(u"Now adding/updating: " + artist['artist_name'])
-    controlValueDict = {"ArtistID":     artistid}
+    controlValueDict = {"ArtistID": artistid}
-    newValueDict = {"ArtistName":       artist['artist_name'],
+    newValueDict = {"ArtistName": artist['artist_name'],
-                    "ArtistSortName":   sortname,
+                    "ArtistSortName": sortname,
-                    "DateAdded":        helpers.today(),
+                    "DateAdded": helpers.today(),
-                    "Status":           "Loading"}
+                    "Status": "Loading"}
 
     myDB.upsert("artists", newValueDict, controlValueDict)
 
@@ -227,7 +226,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
         rgid = rg['id']
         skip_log = 0
         #Make a user configurable variable to skip update of albums with release dates older than this date (in days)
-        pause_delta = headphones.MB_IGNORE_AGE
+        pause_delta = headphones.CONFIG.MB_IGNORE_AGE
 
         rg_exists = myDB.action("SELECT * from albums WHERE AlbumID=?", [rg['id']]).fetchone()
 
@@ -240,27 +239,26 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                 check_release_date = None
                 new_release_group = True
 
-
             if new_release_group:
                 logger.info("[%s] Now adding: %s (New Release Group)" % (artist['artist_name'], rg['title']))
-                new_releases = mb.get_new_releases(rgid,includeExtras)
+                new_releases = mb.get_new_releases(rgid, includeExtras)
 
             else:
                 if check_release_date is None or check_release_date == u"None":
                     logger.info("[%s] Now updating: %s (No Release Date)" % (artist['artist_name'], rg['title']))
-                    new_releases = mb.get_new_releases(rgid,includeExtras,True)
+                    new_releases = mb.get_new_releases(rgid, includeExtras, True)
                 else:
                     if len(check_release_date) == 10:
                         release_date = check_release_date
                     elif len(check_release_date) == 7:
-                        release_date = check_release_date+"-31"
+                        release_date = check_release_date + "-31"
                     elif len(check_release_date) == 4:
-                        release_date = check_release_date+"-12-31"
+                        release_date = check_release_date + "-12-31"
                     else:
                         release_date = today
                     if helpers.get_age(today) - helpers.get_age(release_date) < pause_delta:
                         logger.info("[%s] Now updating: %s (Release Date <%s Days)", artist['artist_name'], rg['title'], pause_delta)
-                        new_releases = mb.get_new_releases(rgid,includeExtras,True)
+                        new_releases = mb.get_new_releases(rgid, includeExtras, True)
                     else:
                         logger.info("[%s] Skipping: %s (Release Date >%s Days)", artist['artist_name'], rg['title'], pause_delta)
                         skip_log = 1
@@ -273,7 +271,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                 new_releases = new_releases
         else:
             logger.info("[%s] Now adding/updating: %s (Comprehensive Force)", artist['artist_name'], rg['title'])
-            new_releases = mb.get_new_releases(rgid,includeExtras,forcefull)
+            new_releases = mb.get_new_releases(rgid, includeExtras, forcefull)
 
         if new_releases != 0:
             # Dump existing hybrid release since we're repackaging/replacing it
@@ -292,26 +290,26 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
             for items in find_hybrid_releases:
                 if items['ReleaseID'] != rg['id']: #don't include hybrid information, since that's what we're replacing
                     hybrid_release_id = items['ReleaseID']
-                    newValueDict = {"ArtistID":         items['ArtistID'],
+                    newValueDict = {"ArtistID": items['ArtistID'],
-                        "ArtistName":       items['ArtistName'],
+                        "ArtistName": items['ArtistName'],
-                        "AlbumTitle":       items['AlbumTitle'],
+                        "AlbumTitle": items['AlbumTitle'],
-                        "AlbumID":          items['AlbumID'],
+                        "AlbumID": items['AlbumID'],
-                        "AlbumASIN":        items['AlbumASIN'],
+                        "AlbumASIN": items['AlbumASIN'],
-                        "ReleaseDate":      items['ReleaseDate'],
+                        "ReleaseDate": items['ReleaseDate'],
-                        "Type":             items['Type'],
+                        "Type": items['Type'],
-                        "ReleaseCountry":   items['ReleaseCountry'],
+                        "ReleaseCountry": items['ReleaseCountry'],
-                        "ReleaseFormat":    items['ReleaseFormat']
+                        "ReleaseFormat": items['ReleaseFormat']
                     }
                     find_hybrid_tracks = myDB.action("SELECT * from alltracks WHERE ReleaseID=?", [hybrid_release_id])
                     totalTracks = 1
                     hybrid_track_array = []
                     for hybrid_tracks in find_hybrid_tracks:
                         hybrid_track_array.append({
-                            'number':        hybrid_tracks['TrackNumber'],
+                            'number': hybrid_tracks['TrackNumber'],
-                            'title':         hybrid_tracks['TrackTitle'],
+                            'title': hybrid_tracks['TrackTitle'],
-                            'id':            hybrid_tracks['TrackID'],
+                            'id': hybrid_tracks['TrackID'],
                             #'url':           hybrid_tracks['TrackURL'],
-                            'duration':      hybrid_tracks['TrackDuration']
+                            'duration': hybrid_tracks['TrackDuration']
                             })
                         totalTracks += 1
                     newValueDict['ReleaseID'] = hybrid_release_id
@@ -325,21 +323,21 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                 logger.info('[%s] Packaging %s releases into hybrid title' % (artist['artist_name'], rg['title']))
             except Exception as e:
                 errors = True
-                logger.warn('[%s] Unable to get hybrid release information for %s: %s' % (artist['artist_name'],rg['title'],e))
+                logger.warn('[%s] Unable to get hybrid release information for %s: %s' % (artist['artist_name'], rg['title'], e))
                 continue
 
             # Use the ReleaseGroupID as the ReleaseID for the hybrid release to differentiate it
             # We can then use the condition WHERE ReleaseID == ReleaseGroupID to select it
             # The hybrid won't have a country or a format
-            controlValueDict = {"ReleaseID":  rg['id']}
+            controlValueDict = {"ReleaseID": rg['id']}
 
-            newValueDict = {"ArtistID":         artistid,
+            newValueDict = {"ArtistID": artistid,
-                            "ArtistName":       artist['artist_name'],
+                            "ArtistName": artist['artist_name'],
-                            "AlbumTitle":       rg['title'],
+                            "AlbumTitle": rg['title'],
-                            "AlbumID":          rg['id'],
+                            "AlbumID": rg['id'],
-                            "AlbumASIN":        hybridrelease['AlbumASIN'],
+                            "AlbumASIN": hybridrelease['AlbumASIN'],
-                            "ReleaseDate":      hybridrelease['ReleaseDate'],
+                            "ReleaseDate": hybridrelease['ReleaseDate'],
-                            "Type":             rg['type']
+                            "Type": rg['type']
                         }
 
             myDB.upsert("allalbums", newValueDict, controlValueDict)
@@ -348,18 +346,18 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
                 cleanname = helpers.cleanName(artist['artist_name'] + ' ' + rg['title'] + ' ' + track['title'])
 
-                controlValueDict = {"TrackID":      track['id'],
+                controlValueDict = {"TrackID": track['id'],
-                                    "ReleaseID":    rg['id']}
+                                    "ReleaseID": rg['id']}
 
-                newValueDict = {"ArtistID":         artistid,
+                newValueDict = {"ArtistID": artistid,
-                                "ArtistName":       artist['artist_name'],
+                                "ArtistName": artist['artist_name'],
-                                "AlbumTitle":       rg['title'],
+                                "AlbumTitle": rg['title'],
-                                "AlbumASIN":        hybridrelease['AlbumASIN'],
+                                "AlbumASIN": hybridrelease['AlbumASIN'],
-                                "AlbumID":          rg['id'],
+                                "AlbumID": rg['id'],
-                                "TrackTitle":       track['title'],
+                                "TrackTitle": track['title'],
-                                "TrackDuration":    track['duration'],
+                                "TrackDuration": track['duration'],
-                                "TrackNumber":      track['number'],
+                                "TrackNumber": track['number'],
-                                "CleanName":        cleanname
+                                "CleanName": cleanname
                             }
 
                 match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?', [cleanname]).fetchone()
@@ -392,35 +390,35 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
             album = myDB.action('SELECT * from allalbums WHERE ReleaseID=?', [releaseid]).fetchone()
 
-            controlValueDict = {"AlbumID":  rg['id']}
+            controlValueDict = {"AlbumID": rg['id']}
 
-            newValueDict = {"ArtistID":         album['ArtistID'],
+            newValueDict = {"ArtistID": album['ArtistID'],
-                            "ArtistName":       album['ArtistName'],
+                            "ArtistName": album['ArtistName'],
-                            "AlbumTitle":       album['AlbumTitle'],
+                            "AlbumTitle": album['AlbumTitle'],
-                            "ReleaseID":        album['ReleaseID'],
+                            "ReleaseID": album['ReleaseID'],
-                            "AlbumASIN":        album['AlbumASIN'],
+                            "AlbumASIN": album['AlbumASIN'],
-                            "ReleaseDate":      album['ReleaseDate'],
+                            "ReleaseDate": album['ReleaseDate'],
-                            "Type":             album['Type'],
+                            "Type": album['Type'],
-                            "ReleaseCountry":   album['ReleaseCountry'],
+                            "ReleaseCountry": album['ReleaseCountry'],
-                            "ReleaseFormat":    album['ReleaseFormat']
+                            "ReleaseFormat": album['ReleaseFormat']
                         }
 
             if rg_exists:
                 newValueDict['DateAdded'] = rg_exists['DateAdded']
-                newValueDict['Status']    = rg_exists['Status']
+                newValueDict['Status'] = rg_exists['Status']
 
             else:
                 today = helpers.today()
 
                 newValueDict['DateAdded'] = today
 
-                if headphones.AUTOWANT_ALL:
+                if headphones.CONFIG.AUTOWANT_ALL:
                     newValueDict['Status'] = "Wanted"
-                elif album['ReleaseDate'] > today and headphones.AUTOWANT_UPCOMING:
+                elif album['ReleaseDate'] > today and headphones.CONFIG.AUTOWANT_UPCOMING:
                     newValueDict['Status'] = "Wanted"
                 # Sometimes "new" albums are added to musicbrainz after their release date, so let's try to catch these
                 # The first test just makes sure we have year-month-day
-                elif helpers.get_age(album['ReleaseDate']) and helpers.get_age(today) - helpers.get_age(album['ReleaseDate']) < 21 and headphones.AUTOWANT_UPCOMING:
+                elif helpers.get_age(album['ReleaseDate']) and helpers.get_age(today) - helpers.get_age(album['ReleaseDate']) < 21 and headphones.CONFIG.AUTOWANT_UPCOMING:
                     newValueDict['Status'] = "Wanted"
                 else:
                     newValueDict['Status'] = "Skipped"
@@ -440,21 +438,21 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                 continue
 
             for track in tracks:
-                controlValueDict = {"TrackID":  track['TrackID'],
+                controlValueDict = {"TrackID": track['TrackID'],
-                                    "AlbumID":  rg['id']}
+                                    "AlbumID": rg['id']}
 
-                newValueDict = {"ArtistID":     track['ArtistID'],
+                newValueDict = {"ArtistID": track['ArtistID'],
-                            "ArtistName":       track['ArtistName'],
+                            "ArtistName": track['ArtistName'],
-                            "AlbumTitle":       track['AlbumTitle'],
+                            "AlbumTitle": track['AlbumTitle'],
-                            "AlbumASIN":        track['AlbumASIN'],
+                            "AlbumASIN": track['AlbumASIN'],
-                            "ReleaseID":        track['ReleaseID'],
+                            "ReleaseID": track['ReleaseID'],
-                            "TrackTitle":       track['TrackTitle'],
+                            "TrackTitle": track['TrackTitle'],
-                            "TrackDuration":    track['TrackDuration'],
+                            "TrackDuration": track['TrackDuration'],
-                            "TrackNumber":      track['TrackNumber'],
+                            "TrackNumber": track['TrackNumber'],
-                            "CleanName":        track['CleanName'],
+                            "CleanName": track['CleanName'],
-                            "Location":         track['Location'],
+                            "Location": track['Location'],
-                            "Format":           track['Format'],
+                            "Format": track['Format'],
-                            "BitRate":          track['BitRate']
+                            "BitRate": track['BitRate']
                             }
 
                 myDB.upsert("tracks", newValueDict, controlValueDict)
@@ -464,11 +462,11 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
             marked_as_downloaded = False
 
             if rg_exists:
-                if rg_exists['Status'] == 'Skipped' and ((have_track_count/float(total_track_count)) >= (headphones.ALBUM_COMPLETION_PCT/100.0)):
+                if rg_exists['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
                     myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
                     marked_as_downloaded = True
             else:
-                if ((have_track_count/float(total_track_count)) >= (headphones.ALBUM_COMPLETION_PCT/100.0)):
+                if (have_track_count / float(total_track_count)) >= (headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0):
                     myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
                     marked_as_downloaded = True
 
@@ -478,7 +476,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
             # Start a search for the album if it's new, hasn't been marked as
             # downloaded and autowant_all is selected. This search is deferred,
             # in case the search failes and the rest of the import will halt.
-            if not rg_exists and not marked_as_downloaded and headphones.AUTOWANT_ALL:
+            if not rg_exists and not marked_as_downloaded and headphones.CONFIG.AUTOWANT_ALL:
                 album_searches.append(rg['id'])
         else:
             if skip_log == 0:
@@ -504,6 +502,7 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
         for album_search in album_searches:
             searcher.searchforalbum(albumid=album_search)
 
+
 def finalize_update(artistid, artistname, errors=False):
     # Moving this little bit to it's own function so we can update have tracks & latest album when deleting extras
 
@@ -514,25 +513,26 @@ def finalize_update(artistid, artistname, errors=False):
     #havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist['artist_name']]))
     havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artistname]))
 
-    controlValueDict = {"ArtistID":     artistid}
+    controlValueDict = {"ArtistID": artistid}
 
     if latestalbum:
-        newValueDict = {"Status":           "Active",
+        newValueDict = {"Status": "Active",
-                        "LatestAlbum":      latestalbum['AlbumTitle'],
+                        "LatestAlbum": latestalbum['AlbumTitle'],
-                        "ReleaseDate":      latestalbum['ReleaseDate'],
+                        "ReleaseDate": latestalbum['ReleaseDate'],
-                        "AlbumID":          latestalbum['AlbumID'],
+                        "AlbumID": latestalbum['AlbumID'],
-                        "TotalTracks":      totaltracks,
+                        "TotalTracks": totaltracks,
-                        "HaveTracks":       havetracks}
+                        "HaveTracks": havetracks}
     else:
-        newValueDict = {"Status":           "Active",
+        newValueDict = {"Status": "Active",
-                        "TotalTracks":      totaltracks,
+                        "TotalTracks": totaltracks,
-                        "HaveTracks":       havetracks}
+                        "HaveTracks": havetracks}
 
     if not errors:
         newValueDict['LastUpdated'] = helpers.now()
 
     myDB.upsert("artists", newValueDict, controlValueDict)
 
+
 def addReleaseById(rid, rgid=None):
 
     myDB = db.DBConnection()
@@ -543,10 +543,10 @@ def addReleaseById(rid, rgid=None):
         dbalbum = myDB.select("SELECT * from albums WHERE AlbumID=?", [rgid])
         if not dbalbum:
             status = 'Loading'
-            controlValueDict = {"AlbumID":  rgid}
+            controlValueDict = {"AlbumID": rgid}
-            newValueDict = {"AlbumTitle":   rgid,
+            newValueDict = {"AlbumTitle": rgid,
-                            "ArtistName":   status,
+                            "ArtistName": status,
-                            "Status":       status}
+                            "Status": status}
             myDB.upsert("albums", newValueDict, controlValueDict)
             time.sleep(1)
 
@@ -590,15 +590,26 @@ def addReleaseById(rid, rgid=None):
             sortname = release_dict['artist_name']
 
         logger.info(u"Now manually adding: " + release_dict['artist_name'] + " - with status Paused")
-        controlValueDict = {"ArtistID":     release_dict['artist_id']}
+        controlValueDict = {"ArtistID": release_dict['artist_id']}
-        newValueDict = {"ArtistName":       release_dict['artist_name'],
+        newValueDict = {"ArtistName": release_dict['artist_name'],
-                        "ArtistSortName":   sortname,
+                        "ArtistSortName": sortname,
-                        "DateAdded":        helpers.today(),
+                        "DateAdded": helpers.today(),
-                        "Status":           "Paused"}
+                        "Status": "Paused"}
 
-        if headphones.INCLUDE_EXTRAS:
+        if headphones.CONFIG.INCLUDE_EXTRAS:
             newValueDict['IncludeExtras'] = 1
-            newValueDict['Extras'] = headphones.EXTRAS
+            newValueDict['Extras'] = headphones.CONFIG.EXTRAS
+
+        if 'title' in release_dict:
+            newValueDict['LatestAlbum'] = release_dict['title']
+        elif 'rg_title' in release_dict:
+            newValueDict['LatestAlbum'] = release_dict['rg_title']
+
+        if 'date' in release_dict:
+            newValueDict['ReleaseDate'] = release_dict['date']
+
+        if rgid:
+            newValueDict['AlbumID'] = rgid
 
         myDB.upsert("artists", newValueDict, controlValueDict)
 
@@ -611,20 +622,20 @@ def addReleaseById(rid, rgid=None):
     if not rg_exists and release_dict or status == 'Loading' and release_dict:  #it should never be the case that we have an rg and not the artist
                                                                                 #but if it is this will fail
         logger.info(u"Now adding-by-id album (" + release_dict['title'] + ") from id: " + rgid)
-        controlValueDict = {"AlbumID":  rgid}
+        controlValueDict = {"AlbumID": rgid}
         if status != 'Loading':
             status = 'Wanted'
 
-        newValueDict = {"ArtistID":         release_dict['artist_id'],
+        newValueDict = {"ArtistID": release_dict['artist_id'],
-                        "ReleaseID":        rgid,
+                        "ReleaseID": rgid,
-                        "ArtistName":       release_dict['artist_name'],
+                        "ArtistName": release_dict['artist_name'],
-                        "AlbumTitle":       release_dict['rg_title'],
+                        "AlbumTitle": release_dict['title'] if 'title' in release_dict else release_dict['rg_title'],
-                        "AlbumASIN":        release_dict['asin'],
+                        "AlbumASIN": release_dict['asin'],
-                        "ReleaseDate":      release_dict['date'],
+                        "ReleaseDate": release_dict['date'],
-                        "DateAdded":        helpers.today(),
+                        "DateAdded": helpers.today(),
-                        "Status":           status,
+                        "Status": status,
-                        "Type":             release_dict['rg_type'],
+                        "Type": release_dict['rg_type'],
-                        "ReleaseID":        rid
+                        "ReleaseID": rid
                         }
 
         myDB.upsert("albums", newValueDict, controlValueDict)
@@ -635,22 +646,22 @@ def addReleaseById(rid, rgid=None):
         for track in release_dict['tracks']:
             cleanname = helpers.cleanName(release_dict['artist_name'] + ' ' + release_dict['rg_title'] + ' ' + track['title'])
 
-            controlValueDict = {"TrackID":  track['id'],
+            controlValueDict = {"TrackID": track['id'],
-                                "AlbumID":  rgid}
+                                "AlbumID": rgid}
-            newValueDict = {"ArtistID":     release_dict['artist_id'],
+            newValueDict = {"ArtistID": release_dict['artist_id'],
-                        "ArtistName":       release_dict['artist_name'],
+                        "ArtistName": release_dict['artist_name'],
-                        "AlbumTitle":       release_dict['rg_title'],
+                        "AlbumTitle": release_dict['rg_title'],
-                        "AlbumASIN":        release_dict['asin'],
+                        "AlbumASIN": release_dict['asin'],
-                        "TrackTitle":       track['title'],
+                        "TrackTitle": track['title'],
-                        "TrackDuration":    track['duration'],
+                        "TrackDuration": track['duration'],
-                        "TrackNumber":      track['number'],
+                        "TrackNumber": track['number'],
-                        "CleanName":        cleanname
+                        "CleanName": cleanname
                         }
 
-            match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?', [cleanname]).fetchone()
+            match = myDB.action('SELECT Location, BitRate, Format, Matched from have WHERE CleanName=?', [cleanname]).fetchone()
 
             if not match:
-                match = myDB.action('SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [release_dict['artist_name'], release_dict['rg_title'], track['title']]).fetchone()
+                match = myDB.action('SELECT Location, BitRate, Format, Matched from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [release_dict['artist_name'], release_dict['rg_title'], track['title']]).fetchone()
 
             #if not match:
                 #match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
@@ -661,17 +672,26 @@ def addReleaseById(rid, rgid=None):
                 newValueDict['Format'] = match['Format']
                 #myDB.action('DELETE from have WHERE Location=?', [match['Location']])
 
+                # If the album has been scanned before adding the release it will be unmatched, update to matched
+                if match['Matched'] == 'Failed':
+                    myDB.action('UPDATE have SET Matched=? WHERE Location=?', (release_dict['rgid'], match['Location']))
+
             myDB.upsert("tracks", newValueDict, controlValueDict)
 
         # Reset status
         if status == 'Loading':
-            controlValueDict = {"AlbumID":  rgid}
+            controlValueDict = {"AlbumID": rgid}
-            newValueDict = {"Status":   "Wanted"}
+            if headphones.CONFIG.AUTOWANT_MANUALLY_ADDED:
+                newValueDict = {"Status": "Wanted"}
+            else:
+                newValueDict = {"Status": "Skipped"}
             myDB.upsert("albums", newValueDict, controlValueDict)
 
         # Start a search for the album
-        import searcher
+        if headphones.CONFIG.AUTOWANT_MANUALLY_ADDED:
-        searcher.searchforalbum(rgid, False)
+            import searcher
+            searcher.searchforalbum(rgid, False)
+
     elif not rg_exists and not release_dict:
         logger.error("ReleaseGroup does not exist in the database and did not get a valid response from MB. Skipping release.")
         if status == 'Loading':
@@ -680,6 +700,7 @@ def addReleaseById(rid, rgid=None):
     else:
         logger.info('Release ' + str(rid) + " already exists in the database!")
 
+
 def updateFormat():
     myDB = db.DBConnection()
     tracks = myDB.select('SELECT * from tracks WHERE Location IS NOT NULL and Format IS NULL')
@@ -688,10 +709,10 @@ def updateFormat():
         for track in tracks:
             try:
                 f = MediaFile(track['Location'])
-            except Exception, e:
+            except Exception as e:
                 logger.info("Exception from MediaFile for: " + track['Location'] + " : " + str(e))
                 continue
-            controlValueDict = {"TrackID":  track['TrackID']}
+            controlValueDict = {"TrackID": track['TrackID']}
             newValueDict = {"Format": f.format}
             myDB.upsert("tracks", newValueDict, controlValueDict)
         logger.info('Finished finding media format for %s files' % len(tracks))
@@ -701,14 +722,15 @@ def updateFormat():
         for track in havetracks:
             try:
                 f = MediaFile(track['Location'])
-            except Exception, e:
+            except Exception as e:
                 logger.info("Exception from MediaFile for: " + track['Location'] + " : " + str(e))
                 continue
-            controlValueDict = {"TrackID":  track['TrackID']}
+            controlValueDict = {"TrackID": track['TrackID']}
             newValueDict = {"Format": f.format}
             myDB.upsert("have", newValueDict, controlValueDict)
         logger.info('Finished finding media format for %s files' % len(havetracks))
 
+
 def getHybridRelease(fullreleaselist):
     """
     Returns a dictionary of best group of tracks from the list of releases and
@@ -721,18 +743,18 @@ def getHybridRelease(fullreleaselist):
     sortable_release_list = []
 
     formats = {
-        '2xVinyl':          '2',
+        '2xVinyl': '2',
-        'Vinyl':            '2',
+        'Vinyl': '2',
-        'CD':               '0',
+        'CD': '0',
-        'Cassette':         '3',
+        'Cassette': '3',
-        '2xCD':             '1',
+        '2xCD': '1',
-        'Digital Media':    '0'
+        'Digital Media': '0'
     }
 
     countries = {
-        'US':    '0',
+        'US': '0',
-        'GB':    '1',
+        'GB': '1',
-        'JP':    '2',
+        'JP': '2',
     }
 
     for release in fullreleaselist:
@@ -749,14 +771,14 @@ def getHybridRelease(fullreleaselist):
 
         # Create record
         release_dict = {
-            'hasasin':        bool(release['AlbumASIN']),
+            'hasasin': bool(release['AlbumASIN']),
-            'asin':           release['AlbumASIN'],
+            'asin': release['AlbumASIN'],
-            'trackscount':    len(release['Tracks']),
+            'trackscount': len(release['Tracks']),
-            'releaseid':      release['ReleaseID'],
+            'releaseid': release['ReleaseID'],
-            'releasedate':    release['ReleaseDate'],
+            'releasedate': release['ReleaseDate'],
-            'format':         format,
+            'format': format,
-            'country':        country,
+            'country': country,
-            'tracks':         release['Tracks']
+            'tracks': release['Tracks']
         }
 
         sortable_release_list.append(release_dict)
@@ -767,8 +789,8 @@ def getHybridRelease(fullreleaselist):
         # Change this value to change the sorting behaviour of none, returning
         # 'None' will put it at the top which was normal behaviour for pre-ngs
         # versions
-        if releaseDate == None:
+        if releaseDate is None:
-            return 'None';
+            return 'None'
 
         if releaseDate.count('-') == 2:
             return releaseDate
@@ -777,7 +799,7 @@ def getHybridRelease(fullreleaselist):
         else:
             return releaseDate + '13-32'
 
-    sortable_release_list.sort(key=lambda x:getSortableReleaseDate(x['releasedate']))
+    sortable_release_list.sort(key=lambda x: getSortableReleaseDate(x['releasedate']))
 
     average_tracks = sum(x['trackscount'] for x in sortable_release_list) / float(len(sortable_release_list))
     for item in sortable_release_list:
@@ -785,9 +807,9 @@ def getHybridRelease(fullreleaselist):
 
     a = helpers.multikeysort(sortable_release_list, ['-hasasin', 'country', 'format', 'trackscount_delta'])
 
-    release_dict = {'ReleaseDate'    : sortable_release_list[0]['releasedate'],
+    release_dict = {'ReleaseDate': sortable_release_list[0]['releasedate'],
-                    'Tracks'         : a[0]['tracks'],
+                    'Tracks': a[0]['tracks'],
-                    'AlbumASIN'      : a[0]['asin']
+                    'AlbumASIN': a[0]['asin']
                     }
 
     return release_dict

headphones/lastfm.py

@@ -14,17 +14,22 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import random
-import time
 import headphones
+import headphones.lock
 
 from headphones import db, logger, request
 
 from collections import defaultdict
 
-TIMEOUT = 60 # seconds
+TIMEOUT = 60.0 # seconds
+REQUEST_LIMIT = 1.0 / 5 # seconds
 ENTRY_POINT = "http://ws.audioscrobbler.com/2.0/"
 API_KEY = "395e6ec6bb557382fc41fde867bce66f"
 
+# Required for API request limit
+lastfm_lock = headphones.lock.TimedLock(REQUEST_LIMIT)
+
+
 def request_lastfm(method, **kwargs):
     """
     Call a Last.FM API method. Automatically sets the method and API key. Method
@@ -43,7 +48,7 @@ def request_lastfm(method, **kwargs):
     logger.debug("Calling Last.FM method: %s", method)
     logger.debug("Last.FM call parameters: %s", kwargs)
 
-    data = request.request_json(ENTRY_POINT, timeout=TIMEOUT, params=kwargs)
+    data = request.request_json(ENTRY_POINT, timeout=TIMEOUT, params=kwargs, lock=lastfm_lock)
 
     # Parse response and check for errors.
     if not data:
@@ -56,6 +61,7 @@ def request_lastfm(method, **kwargs):
 
     return data
 
+
 def getSimilar():
     myDB = db.DBConnection()
     results = myDB.select("SELECT ArtistID from artists ORDER BY HaveTracks DESC")
@@ -65,7 +71,6 @@ def getSimilar():
 
     for result in results[:12]:
         data = request_lastfm("artist.getsimilar", mbid=result["ArtistId"])
-        time.sleep(10)
 
         if data and "similarartists" in data:
             artists = data["similarartists"]["artist"]
@@ -101,16 +106,17 @@ def getSimilar():
 
     logger.debug("Inserted %d artists into Last.FM tag cloud", len(top_list))
 
+
 def getArtists():
     myDB = db.DBConnection()
     results = myDB.select("SELECT ArtistID from artists")
 
-    if not headphones.LASTFM_USERNAME:
+    if not headphones.CONFIG.LASTFM_USERNAME:
         logger.warn("Last.FM username not set, not importing artists.")
         return
 
-    logger.info("Fetching artists from Last.FM for username: %s", headphones.LASTFM_USERNAME)
+    logger.info("Fetching artists from Last.FM for username: %s", headphones.CONFIG.LASTFM_USERNAME)
-    data = request_lastfm("library.getartists", limit=10000, user=headphones.LASTFM_USERNAME)
+    data = request_lastfm("library.getartists", limit=10000, user=headphones.CONFIG.LASTFM_USERNAME)
 
     if data and "artists" in data:
         artistlist = []
@@ -130,6 +136,7 @@ def getArtists():
 
         logger.info("Imported %d new artists from Last.FM", len(artistlist))
 
+
 def getTagTopArtists(tag, limit=50):
     myDB = db.DBConnection()
     results = myDB.select("SELECT ArtistID from artists")
@@ -153,4 +160,4 @@ def getTagTopArtists(tag, limit=50):
         for artistid in artistlist:
             importer.addArtisttoDB(artistid)
 
-        logger.debug("Added %d new artists from Last.FM", len(artistlist))
+        logger.debug("Added %d new artists from Last.FM", len(artistlist))

headphones/librarysync.py

@@ -14,25 +14,25 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import glob
 import headphones
 
 from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
 
 from headphones import db, logger, helpers, importer, lastfm
 
-# You can scan a single directory and append it to the current library by specifying append=True, ArtistID & ArtistName
+# You can scan a single directory and append it to the current library by
-def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=False):
+# specifying append=True, ArtistID and ArtistName.
+def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
+    cron=False):
 
-
+    if cron and not headphones.CONFIG.LIBRARYSCAN:
-    if cron and not headphones.LIBRARYSCAN:
         return
 
     if not dir:
-        if not headphones.MUSIC_DIR:
+        if not headphones.CONFIG.MUSIC_DIR:
             return
         else:
-            dir = headphones.MUSIC_DIR
+            dir = headphones.CONFIG.MUSIC_DIR
 
     # If we're appending a dir, it's coming from the post processor which is
     # already bytestring
@@ -78,22 +78,21 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
 
     latest_subdirectory = []
 
-    for r,d,f in os.walk(dir):
+    for r, d, f in helpers.walk_directory(dir):
-        #need to abuse slicing to get a copy of the list, doing it directly will skip the element after a deleted one
+        # Filter paths based on config. Note that these methods work directly
-        #using a list comprehension will not work correctly for nested subdirectories (os.walk keeps its original list)
+        # on the inputs
-        for directory in d[:]:
+        helpers.path_filter_patterns(d, headphones.CONFIG.IGNORED_FOLDERS, r)
-            if directory.startswith("."):
+        helpers.path_filter_patterns(f, headphones.CONFIG.IGNORED_FILES, r)
-                d.remove(directory)
 
         for files in f:
             # MEDIA_FORMATS = music file extensions, e.g. mp3, flac, etc
             if any(files.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
-
+                subdirectory = r.replace(dir, '')
-                subdirectory = r.replace(dir,'')
                 latest_subdirectory.append(subdirectory)
-                if file_count == 0 and r.replace(dir,'') !='':
+
+                if file_count == 0 and r.replace(dir, '') != '':
                     logger.info("[%s] Now scanning subdirectory %s" % (dir.decode(headphones.SYS_ENCODING, 'replace'), subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
-                elif latest_subdirectory[file_count] != latest_subdirectory[file_count-1] and file_count !=0:
+                elif latest_subdirectory[file_count] != latest_subdirectory[file_count - 1] and file_count != 0:
                     logger.info("[%s] Now scanning subdirectory %s" % (dir.decode(headphones.SYS_ENCODING, 'replace'), subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
 
                 song = os.path.join(r, files)
@@ -105,7 +104,10 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
                 try:
                     f = MediaFile(song)
                 except (FileTypeError, UnreadableFileError):
-                    logger.error("Cannot read file media file '%s'. It may be corrupted or not a media file.", unicode_song_path)
+                    logger.warning("Cannot read media file '%s', skipping. It may be corrupted or not a media file.", unicode_song_path)
+                    continue
+                except IOError:
+                    logger.warning("Cannnot read media file '%s', skipping. Does the file exists?", unicode_song_path)
                     continue
 
                 # Grab the bitrates for the auto detect bit rate option
@@ -124,24 +126,24 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
                 # TODO: skip adding songs without the minimum requisite information (just a matter of putting together the right if statements)
 
                 if f_artist and f.album and f.title:
-                    CleanName = helpers.cleanName(f_artist +' '+ f.album +' '+ f.title)
+                    CleanName = helpers.cleanName(f_artist + ' ' + f.album + ' ' + f.title)
                 else:
                     CleanName = None
 
-                controlValueDict = {'Location'   : unicode_song_path}
+                controlValueDict = {'Location': unicode_song_path}
 
-                newValueDict = { 'TrackID' : f.mb_trackid,
+                newValueDict = {'TrackID': f.mb_trackid,
                               #'ReleaseID' : f.mb_albumid,
-                              'ArtistName' : f_artist,
+                              'ArtistName': f_artist,
-                              'AlbumTitle' : f.album,
+                              'AlbumTitle': f.album,
                               'TrackNumber': f.track,
                               'TrackLength': f.length,
-                              'Genre'      : f.genre,
+                              'Genre': f.genre,
-                              'Date'       : f.date,
+                              'Date': f.date,
-                              'TrackTitle' : f.title,
+                              'TrackTitle': f.title,
-                              'BitRate'    : f.bitrate,
+                              'BitRate': f.bitrate,
-                              'Format'     : f.format,
+                              'Format': f.format,
-                              'CleanName'  : CleanName
+                              'CleanName': CleanName
                               }
 
                 #song_list.append(song_dict)
@@ -152,7 +154,7 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
                     if f_artist:
                         new_artists.append(f_artist)
                     myDB.upsert("have", newValueDict, controlValueDict)
-                    new_song_count+=1
+                    new_song_count += 1
                 else:
                     if check_exist_song['ArtistName'] != f_artist or check_exist_song['AlbumTitle'] != f.album or check_exist_song['TrackTitle'] != f.title:
                         #Important track metadata has been modified, need to run matcher again
@@ -167,19 +169,18 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
                         myDB.upsert("have", newValueDict, controlValueDict)
                         myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, unicode_song_path])
                         myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, unicode_song_path])
-                        new_song_count+=1
+                        new_song_count += 1
                     else:
                         #This track information hasn't changed
                         if f_artist and check_exist_song['Matched'] != "Ignored":
                             new_artists.append(f_artist)
 
-                file_count+=1
+                file_count += 1
-
 
     # Now we start track matching
     logger.info("%s new/modified songs found and added to the database" % new_song_count)
-    song_list = myDB.action("SELECT * FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace')+"%"])
+    song_list = myDB.action("SELECT * FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"])
-    total_number_of_songs = myDB.action("SELECT COUNT(*) FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace')+"%"]).fetchone()[0]
+    total_number_of_songs = myDB.action("SELECT COUNT(*) FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"]).fetchone()[0]
     logger.info("Found " + str(total_number_of_songs) + " new/modified tracks in: '" + dir.decode(headphones.SYS_ENCODING, 'replace') + "'. Matching tracks to the appropriate releases....")
 
     # Sort the song_list by most vague (e.g. no trackid or releaseid) to most specific (both trackid & releaseid)
@@ -197,14 +198,13 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
         latest_artist.append(song['ArtistName'])
         if song_count == 0:
             logger.info("Now matching songs by %s" % song['ArtistName'])
-        elif latest_artist[song_count] != latest_artist[song_count-1] and song_count !=0:
+        elif latest_artist[song_count] != latest_artist[song_count - 1] and song_count != 0:
             logger.info("Now matching songs by %s" % song['ArtistName'])
 
-        #print song['ArtistName']+' - '+song['AlbumTitle']+' - '+song['TrackTitle']
         song_count += 1
-        completion_percentage = float(song_count)/total_number_of_songs * 100
+        completion_percentage = float(song_count) / total_number_of_songs * 100
 
-        if completion_percentage%10 == 0:
+        if completion_percentage % 10 == 0:
             logger.info("Track matching is " + str(completion_percentage) + "% complete")
 
         #THE "MORE-SPECIFIC" CLAUSES HERE HAVE ALL BEEN REMOVED.  WHEN RUNNING A LIBRARY SCAN, THE ONLY CLAUSES THAT
@@ -217,79 +217,78 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
             track = myDB.action('SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from tracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
             have_updated = False
             if track:
-                controlValueDict = { 'ArtistName' : track['ArtistName'],
+                controlValueDict = {'ArtistName': track['ArtistName'],
-                                     'AlbumTitle' : track['AlbumTitle'],
+                                     'AlbumTitle': track['AlbumTitle'],
-                                     'TrackTitle' : track['TrackTitle'] }
+                                     'TrackTitle': track['TrackTitle']}
-                newValueDict = { 'Location' : song['Location'],
+                newValueDict = {'Location': song['Location'],
-                                 'BitRate'  : song['BitRate'],
+                                 'BitRate': song['BitRate'],
-                                 'Format'   : song['Format'] }
+                                 'Format': song['Format']}
                 myDB.upsert("tracks", newValueDict, controlValueDict)
 
-                controlValueDict2 = { 'Location' : song['Location']}
+                controlValueDict2 = {'Location': song['Location']}
-                newValueDict2 = { 'Matched' : track['AlbumID']}
+                newValueDict2 = {'Matched': track['AlbumID']}
                 myDB.upsert("have", newValueDict2, controlValueDict2)
                 have_updated = True
             else:
                 track = myDB.action('SELECT CleanName, AlbumID from tracks WHERE CleanName LIKE ?', [song['CleanName']]).fetchone()
                 if track:
-                    controlValueDict = { 'CleanName' : track['CleanName']}
+                    controlValueDict = {'CleanName': track['CleanName']}
-                    newValueDict = { 'Location' : song['Location'],
+                    newValueDict = {'Location': song['Location'],
-                                     'BitRate'  : song['BitRate'],
+                                     'BitRate': song['BitRate'],
-                                     'Format'   : song['Format'] }
+                                     'Format': song['Format']}
                     myDB.upsert("tracks", newValueDict, controlValueDict)
 
-                    controlValueDict2 = { 'Location' : song['Location']}
+                    controlValueDict2 = {'Location': song['Location']}
-                    newValueDict2 = { 'Matched' : track['AlbumID']}
+                    newValueDict2 = {'Matched': track['AlbumID']}
                     myDB.upsert("have", newValueDict2, controlValueDict2)
                     have_updated = True
                 else:
-                    controlValueDict2 = { 'Location' : song['Location']}
+                    controlValueDict2 = {'Location': song['Location']}
-                    newValueDict2 = { 'Matched' : "Failed"}
+                    newValueDict2 = {'Matched': "Failed"}
                     myDB.upsert("have", newValueDict2, controlValueDict2)
                     have_updated = True
 
             alltrack = myDB.action('SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from alltracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
             if alltrack:
-                controlValueDict = { 'ArtistName' : alltrack['ArtistName'],
+                controlValueDict = {'ArtistName': alltrack['ArtistName'],
-                                     'AlbumTitle' : alltrack['AlbumTitle'],
+                                     'AlbumTitle': alltrack['AlbumTitle'],
-                                     'TrackTitle' : alltrack['TrackTitle'] }
+                                     'TrackTitle': alltrack['TrackTitle']}
-                newValueDict = { 'Location' : song['Location'],
+                newValueDict = {'Location': song['Location'],
-                                 'BitRate'  : song['BitRate'],
+                                 'BitRate': song['BitRate'],
-                                 'Format'   : song['Format'] }
+                                 'Format': song['Format']}
                 myDB.upsert("alltracks", newValueDict, controlValueDict)
 
-                controlValueDict2 = { 'Location' : song['Location']}
+                controlValueDict2 = {'Location': song['Location']}
-                newValueDict2 = { 'Matched' : alltrack['AlbumID']}
+                newValueDict2 = {'Matched': alltrack['AlbumID']}
                 myDB.upsert("have", newValueDict2, controlValueDict2)
             else:
                 alltrack = myDB.action('SELECT CleanName, AlbumID from alltracks WHERE CleanName LIKE ?', [song['CleanName']]).fetchone()
                 if alltrack:
-                    controlValueDict = { 'CleanName' : alltrack['CleanName']}
+                    controlValueDict = {'CleanName': alltrack['CleanName']}
-                    newValueDict = { 'Location' : song['Location'],
+                    newValueDict = {'Location': song['Location'],
-                                     'BitRate'  : song['BitRate'],
+                                     'BitRate': song['BitRate'],
-                                     'Format'   : song['Format'] }
+                                     'Format': song['Format']}
                     myDB.upsert("alltracks", newValueDict, controlValueDict)
 
-                    controlValueDict2 = { 'Location' : song['Location']}
+                    controlValueDict2 = {'Location': song['Location']}
-                    newValueDict2 = { 'Matched' : alltrack['AlbumID']}
+                    newValueDict2 = {'Matched': alltrack['AlbumID']}
                     myDB.upsert("have", newValueDict2, controlValueDict2)
                 else:
                     # alltracks may not exist if adding album manually, have should only be set to failed if not already updated in tracks
                     if not have_updated:
-                        controlValueDict2 = { 'Location' : song['Location']}
+                        controlValueDict2 = {'Location': song['Location']}
-                        newValueDict2 = { 'Matched' : "Failed"}
+                        newValueDict2 = {'Matched': "Failed"}
                         myDB.upsert("have", newValueDict2, controlValueDict2)
 
         else:
-            controlValueDict2 = { 'Location' : song['Location']}
+            controlValueDict2 = {'Location': song['Location']}
-            newValueDict2 = { 'Matched' : "Failed"}
+            newValueDict2 = {'Matched': "Failed"}
             myDB.upsert("have", newValueDict2, controlValueDict2)
 
         #######myDB.action('INSERT INTO have (ArtistName, AlbumTitle, TrackNumber, TrackTitle, TrackLength, BitRate, Genre, Date, TrackID, Location, CleanName, Format) VALUES( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', [song['ArtistName'], song['AlbumTitle'], song['TrackNumber'], song['TrackTitle'], song['TrackLength'], song['BitRate'], song['Genre'], song['Date'], song['TrackID'], song['Location'], CleanName, song['Format']])
 
     logger.info('Completed matching tracks from directory: %s' % dir.decode(headphones.SYS_ENCODING, 'replace'))
 
-
     if not append:
         logger.info('Updating scanned artist track counts')
 
@@ -298,15 +297,30 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
         current_artists = myDB.select('SELECT ArtistName, ArtistID from artists')
 
         #There was a bug where artists with special characters (-,') would show up in new artists.
-        artist_list = [f for f in unique_artists if helpers.cleanName(f).lower() not in [helpers.cleanName(x[0]).lower() for x in current_artists]]
+        artist_list = [
-        artists_checked = [f for f in unique_artists if helpers.cleanName(f).lower() in [helpers.cleanName(x[0]).lower() for x in current_artists]]
+            x for x in unique_artists
+            if helpers.cleanName(x).lower() not in [
+                helpers.cleanName(y[0]).lower()
+                for y in current_artists
+            ]
+        ]
+        artists_checked = [
+            x for x in unique_artists
+            if helpers.cleanName(x).lower() in [
+                helpers.cleanName(y[0]).lower()
+                for y in current_artists
+            ]
+        ]
 
         # Update track counts
 
         for artist in artists_checked:
             # Have tracks are selected from tracks table and not all tracks because of duplicates
             # We update the track count upon an album switch to compliment this
-            havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL', [artist])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artist]))
+            havetracks = (
+                len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL', [artist]))
+                + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artist]))
+            )
             #Note, some people complain about having "artist have tracks" > # of tracks total in artist official releases
             # (can fix by getting rid of second len statement)
             myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistName=?', [havetracks, artist])
@@ -314,7 +328,7 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
         logger.info('Found %i new artists' % len(artist_list))
 
         if len(artist_list):
-            if headphones.ADD_ARTISTS:
+            if headphones.CONFIG.AUTO_ADD_ARTISTS:
                 logger.info('Importing %i new artists' % len(artist_list))
                 importer.artistlist_to_mbids(artist_list)
             else:
@@ -323,8 +337,8 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
                 for artist in artist_list:
                     myDB.action('INSERT OR IGNORE INTO newartists VALUES (?)', [artist])
 
-        if headphones.DETECT_BITRATE:
+        if headphones.CONFIG.DETECT_BITRATE:
-            headphones.PREFERRED_BITRATE = sum(bitrates)/len(bitrates)/1000
+            headphones.CONFIG.PREFERRED_BITRATE = sum(bitrates) / len(bitrates) / 1000
 
     else:
         # If we're appending a new album to the database, update the artists total track counts
@@ -336,9 +350,12 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
     if not append:
         update_album_status()
         lastfm.getSimilar()
+
     logger.info('Library scan complete')
 
 #ADDED THIS SECTION TO MARK ALBUMS AS DOWNLOADED IF ARTISTS ARE ADDED EN MASSE BEFORE LIBRARY IS SCANNED
+
+
 def update_album_status(AlbumID=None):
     myDB = db.DBConnection()
     logger.info('Counting matched tracks to mark albums as skipped/downloaded')
@@ -351,16 +368,16 @@ def update_album_status(AlbumID=None):
         total_tracks = 0
         have_tracks = 0
         for track in track_counter:
-            total_tracks+=1
+            total_tracks += 1
             if track['Location']:
-                have_tracks+=1
+                have_tracks += 1
         if total_tracks != 0:
             album_completion = float(float(have_tracks) / float(total_tracks)) * 100
         else:
             album_completion = 0
             logger.info('Album %s does not have any tracks in database' % album['AlbumTitle'])
 
-        if album_completion >= headphones.ALBUM_COMPLETION_PCT and album['Status'] == 'Skipped':
+        if album_completion >= headphones.CONFIG.ALBUM_COMPLETION_PCT and album['Status'] == 'Skipped':
             new_album_status = "Downloaded"
 
         # I don't think we want to change Downloaded->Skipped.....
@@ -375,7 +392,7 @@ def update_album_status(AlbumID=None):
         else:
             new_album_status = album['Status']
 
-        myDB.upsert("albums", {'Status'   : new_album_status}, {'AlbumID'   : album['AlbumID']})
+        myDB.upsert("albums", {'Status': new_album_status}, {'AlbumID': album['AlbumID']})
         if new_album_status != album['Status']:
             logger.info('Album %s changed to %s' % (album['AlbumTitle'], new_album_status))
     logger.info('Album status update complete')

headphones/lock.py

@@ -0,0 +1,88 @@
+"""
+Locking-related classes
+"""
+
+import headphones.logger
+import time
+import threading
+import Queue
+
+
+class TimedLock(object):
+    """
+    Enforce request rate limit if applicable. This uses the lock so there
+    is synchronized access to the API. When N threads enter this method, the
+    first will pass trough, since there there was no last request recorded.
+    The last request time will be set. Then, the second thread will unlock,
+    and see that the last request was X seconds ago. It will sleep
+    (request_limit - X) seconds, and then continue. Then the third one will
+    unblock, and so on. After all threads finish, the total time will at
+    least be (N * request_limit) seconds. If some request takes longer than
+    request_limit seconds, the next unblocked thread will wait less.
+    """
+
+    def __init__(self, minimum_delta=0):
+        """
+        Set up the lock
+        """
+        self.lock = threading.Lock()
+        self.last_used = 0
+        self.minimum_delta = minimum_delta
+        self.queue = Queue.Queue()
+
+    def __enter__(self):
+        """
+        Called when with lock: is invoked
+        """
+        self.lock.acquire()
+        delta = time.time() - self.last_used
+        sleep_amount = self.minimum_delta - delta
+        if sleep_amount >= 0:
+            # zero sleeps give the cpu a chance to task-switch
+            headphones.logger.info('Sleeping %s (interval)', sleep_amount)
+            time.sleep(sleep_amount)
+        while not self.queue.empty():
+            try:
+                seconds = self.queue.get(False)
+                headphones.logger.info('Sleeping %s (queued)', seconds)
+                time.sleep(seconds)
+            except Queue.Empty:
+                continue
+            self.queue.task_done()
+
+    def __exit__(self, type, value, traceback):
+        """
+        Called when exiting the with block.
+        """
+        self.last_used = time.time()
+        self.lock.release()
+
+    def snooze(self, seconds):
+        """
+        Asynchronously add time to the next request.
+        Can be called outside
+        of the lock context, but it is possible for the next lock holder
+        to not check the queue until after something adds time to it.
+        """
+        # we use a queue so that we don't have to synchronize
+        # across threads and with or without locks
+        headphones.logger.info('Adding %s to queue', seconds)
+        self.queue.add(seconds)
+
+
+class FakeLock(object):
+    """
+    If no locking or request throttling is needed, use this
+    """
+
+    def __enter__(self):
+        """
+        Do nothing on enter
+        """
+        pass
+
+    def __exit__(self, type, value, traceback):
+        """
+        Do nothing on exit
+        """
+        pass

headphones/logger.py

@@ -24,6 +24,7 @@ import headphones
 import threading
 import traceback
 import logging
+import errno
 import sys
 import os
 
@@ -35,8 +36,8 @@ MAX_FILES = 5
 # Headphones logger
 logger = logging.getLogger("headphones")
 
-# Global queue of multiprocessing logging
+# Global queue for multiprocessing logging
-queue = multiprocessing.Queue()
+queue = None
 
 class LogListHandler(logging.Handler):
     """
@@ -49,6 +50,7 @@ class LogListHandler(logging.Handler):
 
         headphones.LOG_LIST.insert(0, (helpers.now(), message, record.levelname, record.threadName))
 
+
 @contextlib.contextmanager
 def listener():
     """
@@ -56,13 +58,34 @@ def listener():
     To be used in a with statement in the main process, for multiprocessing.
     """
 
-    queue_listener = QueueListener(queue, *logger.handlers)
+    global queue
 
-    try:
+    # Initialize queue if not already done
-        queue_listener.start()
+    if queue is None:
+        try:
+            queue = multiprocessing.Queue()
+        except OSError as e:
+            queue = False
+
+            # Some machines don't have access to /dev/shm. See
+            # http://stackoverflow.com/questions/2009278 for more information.
+            if e.errno == errno.EACCES:
+                logger.warning("Multiprocess logging disabled, because "
+                    "current user cannot map shared memory. You won't see any" \
+                    "logging generated by the worker processed.")
+
+    # Multiprocess logging may be disabled.
+    if not queue:
         yield
-    finally:
+    else:
-        queue_listener.stop()
+        queue_listener = QueueListener(queue, *logger.handlers)
+
+        try:
+            queue_listener.start()
+            yield
+        finally:
+            queue_listener.stop()
+
 
 def initMultiprocessing():
     """
@@ -71,6 +94,11 @@ def initMultiprocessing():
     completely.
     """
 
+    # Multiprocess logging may be disabled.
+    if not queue:
+        return
+
+    # Remove all handlers and add the Queue handler as the only one.
     for handler in logger.handlers[:]:
         logger.removeHandler(handler)
 
@@ -82,7 +110,8 @@ def initMultiprocessing():
     # Change current thread name for log record
     threading.current_thread().name = multiprocessing.current_process().name
 
-def initLogger(console=False, verbose=False):
+
+def initLogger(console=False, log_dir=False, verbose=False):
     """
     Setup logging for Headphones. It uses the logger instance with the name
     'headphones'. Three log handlers are added:
@@ -91,7 +120,8 @@ def initLogger(console=False, verbose=False):
     * LogListHandler: for Web UI
     * StreamHandler: for console (if console)
 
-    Console logging is only enabled if console is set to True.
+    Console logging is only enabled if console is set to True. This method can
+    be invoked multiple times, during different stages of Headphones.
     """
 
     # Close and remove old handlers. This is required to reinit the loggers
@@ -109,22 +139,23 @@ def initLogger(console=False, verbose=False):
     logger.propagate = False
     logger.setLevel(logging.DEBUG if verbose else logging.INFO)
 
-    # Setup file logger
-    filename = os.path.join(headphones.LOG_DIR, FILENAME)
-
-    file_formatter = logging.Formatter('%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
-    file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE, backupCount=MAX_FILES)
-    file_handler.setLevel(logging.DEBUG)
-    file_handler.setFormatter(file_formatter)
-
-    logger.addHandler(file_handler)
-
     # Add list logger
     loglist_handler = LogListHandler()
     loglist_handler.setLevel(logging.DEBUG)
 
     logger.addHandler(loglist_handler)
 
+    # Setup file logger
+    if log_dir:
+        filename = os.path.join(log_dir, FILENAME)
+
+        file_formatter = logging.Formatter('%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
+        file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE, backupCount=MAX_FILES)
+        file_handler.setLevel(logging.DEBUG)
+        file_handler.setFormatter(file_formatter)
+
+        logger.addHandler(file_handler)
+
     # Setup console logger
     if console:
         console_formatter = logging.Formatter('%(asctime)s - %(levelname)s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
@@ -137,6 +168,7 @@ def initLogger(console=False, verbose=False):
     # Install exception hooks
     initHooks()
 
+
 def initHooks(global_exceptions=True, thread_exceptions=True, pass_original=True):
     """
     This method installs exception catching mechanisms. Any exception caught
@@ -191,4 +223,4 @@ warn = logger.warn
 error = logger.error
 debug = logger.debug
 warning = logger.warning
-exception = logger.exception
+exception = logger.exception

headphones/lyrics.py

@@ -18,9 +18,10 @@ import htmlentitydefs
 
 from headphones import logger, request
 
+
 def getLyrics(artist, song):
 
-    params = {  "artist": artist.encode('utf-8'),
+    params = {"artist": artist.encode('utf-8'),
                 "song": song.encode('utf-8'),
                 "fmt": 'xml'
                 }
@@ -60,6 +61,7 @@ def getLyrics(artist, song):
 
     return lyrics
 
+
 def convert_html_entities(s):
     matches = re.findall("&#\d+;", s)
     if len(matches) > 0:
@@ -79,7 +81,7 @@ def convert_html_entities(s):
         hits.remove(amp)
     for hit in hits:
         name = hit[1:-1]
-        if htmlentitydefs.name2codepoint.has_key(name):
+        if name in htmlentitydefs.name2codepoint:
-                s = s.replace(hit, unichr(htmlentitydefs.name2codepoint[name]))
+            s = s.replace(hit, unichr(htmlentitydefs.name2codepoint[name]))
     s = s.replace(amp, "&")
     return s

headphones/mb.py

@@ -15,404 +15,397 @@
 
 
 from headphones import logger, db, helpers
-from headphones.helpers import multikeysort, replace_all
 
-import time
-import threading
 import headphones
 import musicbrainzngs
+import headphones.lock
 
 try:
+    # pylint:disable=E0611
+    # ignore this error because we are catching the ImportError
     from collections import OrderedDict
+    # pylint:enable=E0611
 except ImportError:
     # Python 2.6.x fallback, from libs
-    import OrderedDict
+    from ordereddict import OrderedDict
 
-mb_lock = threading.Lock()
+mb_lock = headphones.lock.TimedLock(0)
 
 # Quick fix to add mirror switching on the fly. Need to probably return the mbhost & mbport that's
 # being used, so we can send those values to the log
+
+
 def startmb():
 
     mbuser = None
     mbpass = None
 
-    if headphones.MIRROR == "musicbrainz.org":
+    if headphones.CONFIG.MIRROR == "musicbrainz.org":
         mbhost = "musicbrainz.org"
         mbport = 80
         sleepytime = 1
-    elif headphones.MIRROR == "custom":
+    elif headphones.CONFIG.MIRROR == "custom":
-        mbhost = headphones.CUSTOMHOST
+        mbhost = headphones.CONFIG.CUSTOMHOST
-        mbport = int(headphones.CUSTOMPORT)
+        mbport = int(headphones.CONFIG.CUSTOMPORT)
-        sleepytime = int(headphones.CUSTOMSLEEP)
+        sleepytime = int(headphones.CONFIG.CUSTOMSLEEP)
-    elif headphones.MIRROR == "headphones":
+    elif headphones.CONFIG.MIRROR == "headphones":
         mbhost = "144.76.94.239"
         mbport = 8181
-        mbuser = headphones.HPUSER
+        mbuser = headphones.CONFIG.HPUSER
-        mbpass = headphones.HPPASS
+        mbpass = headphones.CONFIG.HPPASS
         sleepytime = 0
     else:
         return False
 
-    musicbrainzngs.set_useragent("headphones","0.0","https://github.com/rembo10/headphones")
+    musicbrainzngs.set_useragent("headphones", "0.0", "https://github.com/rembo10/headphones")
     musicbrainzngs.set_hostname(mbhost + ":" + str(mbport))
+
+    # Their rate limiting should be redundant to our lock
     if sleepytime == 0:
         musicbrainzngs.set_rate_limit(False)
     else:
         #calling it with an it ends up blocking all requests after the first
         musicbrainzngs.set_rate_limit(limit_or_interval=float(sleepytime))
+        mb_lock.minimum_delta = sleepytime
 
     # Add headphones credentials
-    if headphones.MIRROR == "headphones":
+    if headphones.CONFIG.MIRROR == "headphones":
         if not mbuser and mbpass:
             logger.warn("No username or password set for VIP server")
         else:
-            musicbrainzngs.hpauth(mbuser,mbpass)
+            musicbrainzngs.hpauth(mbuser, mbpass)
 
     logger.debug('Using the following server values: MBHost: %s, MBPort: %i, Sleep Interval: %i', mbhost, mbport, sleepytime)
 
     return True
 
+
 def findArtist(name, limit=1):
+     artistlist = []
+     artistResults = None
 
-    with mb_lock:
+     chars = set('!?*-')
-        artistlist = []
+     if any((c in chars) for c in name):
-        artistResults = None
+         name = '"' + name + '"'
 
-        chars = set('!?*-')
+     criteria = {'artist': name.lower()}
-        if any((c in chars) for c in name):
-            name = '"'+name+'"'
 
-        criteria = {'artist': name.lower()}
+     with mb_lock:
+         try:
+             artistResults = musicbrainzngs.search_artists(limit=limit, **criteria)['artist-list']
+         except musicbrainzngs.WebServiceError as e:
+             logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
+             mb_lock.snooze(5)
 
-        try:
+     if not artistResults:
-            artistResults = musicbrainzngs.search_artists(limit=limit, **criteria)['artist-list']
+         return False
-        except musicbrainzngs.WebServiceError as e:
+     for result in artistResults:
-            logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
+         if 'disambiguation' in result:
-            time.sleep(5)
+             uniquename = unicode(result['sort-name'] + " (" + result['disambiguation'] + ")")
+         else:
+             uniquename = unicode(result['sort-name'])
+         if result['name'] != uniquename and limit == 1:
+             logger.info('Found an artist with a disambiguation: %s - doing an album based search' % name)
+             artistdict = findArtistbyAlbum(name)
+             if not artistdict:
+                 logger.info('Cannot determine the best match from an artist/album search. Using top match instead')
+                 artistlist.append({
+                 # Just need the artist id if the limit is 1
+                 #    'name':             unicode(result['sort-name']),
+                 #    'uniquename':        uniquename,
+                     'id': unicode(result['id']),
+                 #    'url':                 unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
+                 #    'score':            int(result['ext:score'])
+                     })
+             else:
+                 artistlist.append(artistdict)
+         else:
+             artistlist.append({
+                     'name': unicode(result['sort-name']),
+                     'uniquename': uniquename,
+                     'id': unicode(result['id']),
+                     'url': unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
+                     'score': int(result['ext:score'])
+                     })
+     return artistlist
 
-        if not artistResults:
-            return False
-        for result in artistResults:
-            if 'disambiguation' in result:
-                uniquename = unicode(result['sort-name'] + " (" + result['disambiguation'] + ")")
-            else:
-                uniquename = unicode(result['sort-name'])
-            if result['name'] != uniquename and limit == 1:
-                logger.info('Found an artist with a disambiguation: %s - doing an album based search' % name)
-                artistdict = findArtistbyAlbum(name)
-                if not artistdict:
-                    logger.info('Cannot determine the best match from an artist/album search. Using top match instead')
-                    artistlist.append({
-                    # Just need the artist id if the limit is 1
-                    #    'name':             unicode(result['sort-name']),
-                    #    'uniquename':        uniquename,
-                        'id':                unicode(result['id']),
-                    #    'url':                 unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
-                    #    'score':            int(result['ext:score'])
-                        })
-                else:
-                    artistlist.append(artistdict)
-            else:
-                artistlist.append({
-                        'name':             unicode(result['sort-name']),
-                        'uniquename':       uniquename,
-                        'id':               unicode(result['id']),
-                        'url':              unicode("http://musicbrainz.org/artist/" + result['id']),#probably needs to be changed
-                        'score':            int(result['ext:score'])
-                        })
-        return artistlist
 
 def findRelease(name, limit=1, artist=None):
+    releaselist = []
+    releaseResults = None
+
+    # additional artist search
+    if not artist and ':' in name:
+        name, artist = name.rsplit(":", 1)
+
+    chars = set('!?*-')
+    if any((c in chars) for c in name):
+        name = '"' + name + '"'
+    if artist and any((c in chars) for c in artist):
+        artist = '"' + artist + '"'
 
     with mb_lock:
-        releaselist = []
-        releaseResults = None
-
-        # additional artist search
-        if not artist and ':' in name:
-            name, artist = name.rsplit(":",1)
-
-        chars = set('!?*-')
-        if any((c in chars) for c in name):
-            name = '"'+name+'"'
-        if artist and any((c in chars) for c in artist):
-            artist = '"'+artist+'"'
-
         try:
-            releaseResults = musicbrainzngs.search_releases(query=name,limit=limit,artist=artist)['release-list']
+            releaseResults = musicbrainzngs.search_releases(query=name, limit=limit, artist=artist)['release-list']
         except musicbrainzngs.WebServiceError as e: #need to update exceptions
             logger.warn('Attempt to query MusicBrainz for "%s" failed: %s' % (name, str(e)))
-            time.sleep(5)
+            mb_lock.snooze(5)
 
-        if not releaseResults:
+    if not releaseResults:
-            return False
+        return False
 
-        for result in releaseResults:
+    for result in releaseResults:
 
-            title = result['title']
+        title = result['title']
-            if 'disambiguation' in result:
+        if 'disambiguation' in result:
-                title += ' (' + result['disambiguation'] + ')'
+            title += ' (' + result['disambiguation'] + ')'
 
-            # Get formats and track counts
+        # Get formats and track counts
-            format_dict = OrderedDict()
+        format_dict = OrderedDict()
-            formats = ''
+        formats = ''
-            tracks = ''
+        tracks = ''
-            if 'medium-list' in result:
+        if 'medium-list' in result:
-                for medium in result['medium-list']:
+            for medium in result['medium-list']:
-                    if 'format' in medium:
+                if 'format' in medium:
-                        format = medium['format']
+                    format = medium['format']
-                        if format not in format_dict:
+                    if format not in format_dict:
-                            format_dict[format] = 0
+                        format_dict[format] = 0
-                        format_dict[format] += 1
+                    format_dict[format] += 1
-                    if 'track-count' in medium:
+                if 'track-count' in medium:
-                        if tracks:
+                    if tracks:
-                            tracks += ' + '
+                        tracks += ' + '
-                        tracks += str(medium['track-count'])
+                    tracks += str(medium['track-count'])
-                for format, count in format_dict.items():
+            for format, count in format_dict.items():
-                    if formats:
+                if formats:
-                        formats += ' + '
+                    formats += ' + '
-                    if count > 1:
+                if count > 1:
-                        formats += str(count) + 'x'
+                    formats += str(count) + 'x'
-                    formats += format
+                formats += format
 
-            rg_type = ''
+        rg_type = ''
-            if 'type' in result['release-group']:
+        if 'type' in result['release-group']:
-                rg_type = result['release-group']['type']
+            rg_type = result['release-group']['type']
-                if rg_type == 'Album' and 'secondary-type-list' in result['release-group']:
+            if rg_type == 'Album' and 'secondary-type-list' in result['release-group']:
-                    secondary_type = result['release-group']['secondary-type-list'][0]
+                secondary_type = result['release-group']['secondary-type-list'][0]
+                if secondary_type != rg_type:
+                    rg_type = secondary_type
+
+        releaselist.append({
+                    'uniquename': unicode(result['artist-credit'][0]['artist']['name']),
+                    'title': unicode(title),
+                    'id': unicode(result['artist-credit'][0]['artist']['id']),
+                    'albumid': unicode(result['id']),
+                    'url': unicode("http://musicbrainz.org/artist/" + result['artist-credit'][0]['artist']['id']),#probably needs to be changed
+                    'albumurl': unicode("http://musicbrainz.org/release/" + result['id']),#probably needs to be changed
+                    'score': int(result['ext:score']),
+                    'date': unicode(result['date']) if 'date' in result else '',
+                    'country': unicode(result['country']) if 'country' in result else '',
+                    'formats': unicode(formats),
+                    'tracks': unicode(tracks),
+                    'rgid': unicode(result['release-group']['id']),
+                    'rgtype': unicode(rg_type)
+                    })
+    return releaselist
+
+
+def getArtist(artistid, extrasonly=False):
+    artist_dict = {}
+    artist = None
+    try:
+        limit = 200
+        with mb_lock:
+            artist = musicbrainzngs.get_artist_by_id(artistid)['artist']
+        newRgs = None
+        artist['release-group-list'] = []
+        while newRgs is None or len(newRgs) >= limit:
+            with mb_lock:
+                newRgs = musicbrainzngs.browse_release_groups(
+                    artistid,
+                    release_type="album",
+                    offset=len(artist['release-group-list']),
+                    limit=limit)
+            newRgs = newRgs['release-group-list']
+            artist['release-group-list'] += newRgs
+    except musicbrainzngs.WebServiceError as e:
+        logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
+        mb_lock.snooze(5)
+    except Exception as e:
+        pass
+
+    if not artist:
+        return False
+
+    artist_dict['artist_name'] = unicode(artist['name'])
+
+    releasegroups = []
+
+    if not extrasonly:
+        for rg in artist['release-group-list']:
+            if "secondary-type-list" in rg.keys(): #only add releases without a secondary type
+                continue
+            releasegroups.append({
+                'title': unicode(rg['title']),
+                'id': unicode(rg['id']),
+                'url': u"http://musicbrainz.org/release-group/" + rg['id'],
+                'type': unicode(rg['type'])
+            })
+
+    # See if we need to grab extras. Artist specific extras take precedence over global option
+    # Global options are set when adding a new artist
+    myDB = db.DBConnection()
+
+    try:
+        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?', [artistid]).fetchone()
+        includeExtras = db_artist['IncludeExtras']
+    except IndexError:
+        includeExtras = False
+
+    if includeExtras:
+
+        # Need to convert extras string from something like '2,5.6' to ['ep','live','remix'] (append new extras to end)
+        if db_artist['Extras']:
+            extras = map(int, db_artist['Extras'].split(','))
+        else:
+            extras = []
+        extras_list = headphones.POSSIBLE_EXTRAS
+
+        includes = []
+
+        i = 1
+        for extra in extras_list:
+            if i in extras:
+                includes.append(extra)
+            i += 1
+
+        for include in includes:
+
+            mb_extras_list = []
+
+            try:
+                limit = 200
+                newRgs = None
+                while newRgs is None or len(newRgs) >= limit:
+                    with mb_lock:
+                        newRgs = musicbrainzngs.browse_release_groups(
+                            artistid, release_type=include, offset=len(mb_extras_list), limit=limit)
+                    newRgs = newRgs['release-group-list']
+                    mb_extras_list += newRgs
+            except musicbrainzngs.WebServiceError as e:
+                logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
+                mb_lock.snooze(5)
+
+            for rg in mb_extras_list:
+                rg_type = rg['type']
+                if rg_type == 'Album' and 'secondary-type-list' in rg:
+                    secondary_type = rg['secondary-type-list'][0]
                     if secondary_type != rg_type:
                         rg_type = secondary_type
 
-            releaselist.append({
-                        'uniquename':        unicode(result['artist-credit'][0]['artist']['name']),
-                        'title':             unicode(title),
-                        'id':                unicode(result['artist-credit'][0]['artist']['id']),
-                        'albumid':           unicode(result['id']),
-                        'url':               unicode("http://musicbrainz.org/artist/" + result['artist-credit'][0]['artist']['id']),#probably needs to be changed
-                        'albumurl':          unicode("http://musicbrainz.org/release/" + result['id']),#probably needs to be changed
-                        'score':             int(result['ext:score']),
-                        'date':              unicode(result['date']) if 'date' in result else '',
-                        'country':           unicode(result['country']) if 'country' in result else '',
-                        'formats':           unicode(formats),
-                        'tracks':            unicode(tracks),
-                        'rgid':              unicode(result['release-group']['id']),
-                        'rgtype':            unicode(rg_type)
-                        })
-        return releaselist
-
-def getArtist(artistid, extrasonly=False):
-
-    with mb_lock:
-        artist_dict = {}
-
-        artist = None
-
-        try:
-            limit = 200
-            artist = musicbrainzngs.get_artist_by_id(artistid)['artist']
-            newRgs = None
-            artist['release-group-list'] = []
-            while newRgs == None or len(newRgs) >= limit:
-                newRgs = musicbrainzngs.browse_release_groups(artistid,release_type="album",offset=len(artist['release-group-list']),limit=limit)['release-group-list']
-                artist['release-group-list'] += newRgs
-        except musicbrainzngs.WebServiceError as e:
-            logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
-            time.sleep(5)
-        except Exception,e:
-            pass
-
-        if not artist:
-            return False
-
-        #if 'disambiguation' in artist:
-        #    uniquename = unicode(artist['sort-name'] + " (" + artist['disambiguation'] + ")")
-        #else:
-        #    uniquename = unicode(artist['sort-name'])
-
-        artist_dict['artist_name'] = unicode(artist['name'])
-
-        # Not using the following values anywhere yet so we don't need to grab them.
-        # Was causing an exception to be raised if they didn't exist.
-        #
-        #artist_dict['artist_sortname'] = unicode(artist['sort-name'])
-        #artist_dict['artist_uniquename'] = uniquename
-        #artist_dict['artist_type'] = unicode(artist['type'])
-
-        #artist_dict['artist_begindate'] = None
-        #artist_dict['artist_enddate'] = None
-        #if 'life-span' in artist:
-        #    if 'begin' in artist['life-span']:
-        #        artist_dict['artist_begindate'] = unicode(artist['life-span']['begin'])
-        #    if 'end' in artist['life-span']:
-        #        artist_dict['artist_enddate'] = unicode(artist['life-span']['end'])
-
-
-        releasegroups = []
-
-        if not extrasonly:
-            for rg in artist['release-group-list']:
-                if "secondary-type-list" in rg.keys(): #only add releases without a secondary type
-                    continue
                 releasegroups.append({
-                            'title':      unicode(rg['title']),
+                    'title': unicode(rg['title']),
-                            'id':         unicode(rg['id']),
+                    'id': unicode(rg['id']),
-                            'url':        u"http://musicbrainz.org/release-group/" + rg['id'],
+                    'url': u"http://musicbrainz.org/release-group/" + rg['id'],
-                            'type':       unicode(rg['type'])
+                    'type': unicode(rg_type)
-                    })
+                })
+    artist_dict['releasegroups'] = releasegroups
+    return artist_dict
 
-        # See if we need to grab extras. Artist specific extras take precedence over global option
-        # Global options are set when adding a new artist
-        myDB = db.DBConnection()
-
-        try:
-            db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?', [artistid]).fetchone()
-            includeExtras = db_artist['IncludeExtras']
-        except IndexError:
-            includeExtras = False
-
-        if includeExtras:
-
-            # Need to convert extras string from something like '2,5.6' to ['ep','live','remix'] (append new extras to end)
-            if db_artist['Extras']:
-                extras = map(int, db_artist['Extras'].split(','))
-            else:
-                extras = []
-            extras_list = ["single", "ep", "compilation", "soundtrack", "live", "remix", "spokenword", "audiobook", "other", "dj-mix", "mixtape/street", "broadcast", "interview", "demo"]
-
-            includes = []
-
-            i = 1
-            for extra in extras_list:
-                if i in extras:
-                    includes.append(extra)
-                i += 1
-
-            for include in includes:
-
-                mb_extras_list = []
-
-                try:
-                    limit = 200
-                    newRgs = None
-                    while newRgs == None or len(newRgs) >= limit:
-                        newRgs = musicbrainzngs.browse_release_groups(artistid,release_type=include,offset=len(mb_extras_list),limit=limit)['release-group-list']
-                        mb_extras_list += newRgs
-                except musicbrainzngs.WebServiceError as e:
-                    logger.warn('Attempt to retrieve artist information from MusicBrainz failed for artistid: %s (%s)' % (artistid, str(e)))
-                    time.sleep(5)
-
-                for rg in mb_extras_list:
-
-                    rg_type = rg['type']
-                    if rg_type == 'Album' and 'secondary-type-list' in rg:
-                        secondary_type = rg['secondary-type-list'][0]
-                        if secondary_type != rg_type:
-                            rg_type = secondary_type
-
-                    releasegroups.append({
-                            'title':        unicode(rg['title']),
-                            'id':           unicode(rg['id']),
-                            'url':          u"http://musicbrainz.org/release-group/" + rg['id'],
-                            'type':         unicode(rg_type)
-                        })
-
-        artist_dict['releasegroups'] = releasegroups
-
-        return artist_dict
 
 def getReleaseGroup(rgid):
     """
     Returns a list of releases in a release group
     """
-    with mb_lock:
+    releaseGroup = None
+    try:
+        with mb_lock:
+            releaseGroup = musicbrainzngs.get_release_group_by_id(
+                rgid, ["artists", "releases", "media", "discids", ])
+            releaseGroup = releaseGroup['release-group']
+    except musicbrainzngs.WebServiceError as e:
+        logger.warn('Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (rgid, str(e)))
+        mb_lock.snooze(5)
 
-        releaselist = []
+    if not releaseGroup:
+        return False
+    else:
+        return releaseGroup['release-list']
 
-        releaseGroup = None
-
-        try:
-            releaseGroup = musicbrainzngs.get_release_group_by_id(rgid,["artists","releases","media","discids",])['release-group']
-        except musicbrainzngs.WebServiceError as e:
-            logger.warn('Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (rgid, str(e)))
-            time.sleep(5)
-
-        if not releaseGroup:
-            return False
-        else:
-            return releaseGroup['release-list']
 
 def getRelease(releaseid, include_artist_info=True):
     """
     Deep release search to get track info
     """
-    with mb_lock:
+    release = {}
+    results = None
 
-        release = {}
+    try:
-        results = None
+        with mb_lock:
-
-        try:
             if include_artist_info:
-                results = musicbrainzngs.get_release_by_id(releaseid,["artists","release-groups","media","recordings"]).get('release')
+                results = musicbrainzngs.get_release_by_id(releaseid, ["artists", "release-groups", "media", "recordings"]).get('release')
             else:
-                results = musicbrainzngs.get_release_by_id(releaseid,["media","recordings"]).get('release')
+                results = musicbrainzngs.get_release_by_id(releaseid, ["media", "recordings"]).get('release')
-        except musicbrainzngs.WebServiceError as e:
+    except musicbrainzngs.WebServiceError as e:
-            logger.warn('Attempt to retrieve information from MusicBrainz for release "%s" failed (%s)' % (releaseid, str(e)))
+        logger.warn('Attempt to retrieve information from MusicBrainz for release "%s" failed (%s)' % (releaseid, str(e)))
-            time.sleep(5)
+        mb_lock.snooze(5)
 
-        if not results:
+    if not results:
-            return False
+        return False
 
-        release['title'] = unicode(results['title'])
+    release['title'] = unicode(results['title'])
-        release['id'] = unicode(results['id'])
+    release['id'] = unicode(results['id'])
-        release['asin'] = unicode(results['asin']) if 'asin' in results else None
+    release['asin'] = unicode(results['asin']) if 'asin' in results else None
-        release['date'] = unicode(results['date']) if 'date' in results else None
+    release['date'] = unicode(results['date']) if 'date' in results else None
-        try:
+    try:
-            release['format'] = unicode(results['medium-list'][0]['format'])
+        release['format'] = unicode(results['medium-list'][0]['format'])
-        except:
+    except:
-            release['format'] = u'Unknown'
+        release['format'] = u'Unknown'
 
-        try:
+    try:
-            release['country'] = unicode(results['country'])
+        release['country'] = unicode(results['country'])
-        except:
+    except:
-            release['country'] = u'Unknown'
+        release['country'] = u'Unknown'
+
+    if include_artist_info:
+
+        if 'release-group' in results:
+            release['rgid'] = unicode(results['release-group']['id'])
+            release['rg_title'] = unicode(results['release-group']['title'])
+            try:
+                release['rg_type'] = unicode(results['release-group']['type'])
+
+                if release['rg_type'] == 'Album' and 'secondary-type-list' in results['release-group']:
+                    secondary_type = unicode(results['release-group']['secondary-type-list'][0])
+                    if secondary_type != release['rg_type']:
+                        release['rg_type'] = secondary_type
+
+            except KeyError:
+                release['rg_type'] = u'Unknown'
+
+        else:
+            logger.warn("Release " + releaseid + "had no ReleaseGroup associated")
+
+        release['artist_name'] = unicode(results['artist-credit'][0]['artist']['name'])
+        release['artist_id'] = unicode(results['artist-credit'][0]['artist']['id'])
+
+    release['tracks'] = getTracksFromRelease(results)
+
+    return release
 
 
-        if include_artist_info:
+def get_new_releases(rgid, includeExtras=False, forcefull=False):
-
-            if 'release-group' in results:
-                release['rgid'] = unicode(results['release-group']['id'])
-                release['rg_title'] = unicode(results['release-group']['title'])
-                try:
-                    release['rg_type'] = unicode(results['release-group']['type'])
-
-                    if release['rg_type'] == 'Album' and 'secondary-type-list' in results['release-group']:
-                        secondary_type = unicode(results['release-group']['secondary-type-list'][0])
-                        if secondary_type != release['rg_type']:
-                            release['rg_type'] = secondary_type
-
-                except KeyError:
-                    release['rg_type'] = u'Unknown'
-
-            else:
-                logger.warn("Release " + releaseid + "had no ReleaseGroup associated")
-
-            release['artist_name'] = unicode(results['artist-credit'][0]['artist']['name'])
-            release['artist_id'] = unicode(results['artist-credit'][0]['artist']['id'])
-
-        release['tracks'] = getTracksFromRelease(results)
-
-        return release
-
-def get_new_releases(rgid,includeExtras=False,forcefull=False):
 
     myDB = db.DBConnection()
     results = []
     try:
         limit = 100
         newResults = None
-        while newResults == None or len(newResults) >= limit:
+        while newResults is None or len(newResults) >= limit:
-            newResults = musicbrainzngs.browse_releases(release_group=rgid,includes=['artist-credits','labels','recordings','release-groups','media'],limit=limit,offset=len(results))
+            with mb_lock:
+                newResults = musicbrainzngs.browse_releases(
+                    release_group=rgid,
+                    includes=['artist-credits', 'labels', 'recordings', 'release-groups', 'media'],
+                    limit=limit,
+                    offset=len(results))
             if 'release-list' not in newResults:
                 break #may want to raise an exception here instead ?
             newResults = newResults['release-list']
@@ -420,7 +413,7 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
 
     except musicbrainzngs.WebServiceError as e:
         logger.warn('Attempt to retrieve information from MusicBrainz for release group "%s" failed (%s)' % (rgid, str(e)))
-        time.sleep(5)
+        mb_lock.snooze(5)
         return False
 
     if not results or len(results) == 0:
@@ -457,8 +450,6 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
 
         release = {}
         rel_id_check = releasedata['id']
-        artistid = unicode(releasedata['artist-credit'][0]['artist']['id'])
-
         album_checker = myDB.action('SELECT * from allalbums WHERE ReleaseID=?', [rel_id_check]).fetchone()
         if not album_checker or forcefull:
             #DELETE all references to this release since we're updating it anyway.
@@ -486,21 +477,20 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
                 logger.warn('Release ' + releasedata['id'] + ' has no Artists associated.')
                 return False
 
-
             release['ReleaseCountry'] = unicode(releasedata['country']) if 'country' in releasedata else u'Unknown'
             #assuming that the list will contain media and that the format will be consistent
             try:
-                additional_medium=''
+                additional_medium = ''
                 for position in releasedata['medium-list']:
                     if position['format'] == releasedata['medium-list'][0]['format']:
                         medium_count = int(position['position'])
                     else:
-                        additional_medium = additional_medium+' + '+position['format']
+                        additional_medium = additional_medium + ' + ' + position['format']
                 if medium_count == 1:
                     disc_number = ''
                 else:
-                    disc_number = str(medium_count)+'x'
+                    disc_number = str(medium_count) + 'x'
-                packaged_medium = disc_number+releasedata['medium-list'][0]['format']+additional_medium
+                packaged_medium = disc_number + releasedata['medium-list'][0]['format'] + additional_medium
                 release['ReleaseFormat'] = unicode(packaged_medium)
             except:
                 release['ReleaseFormat'] = u'Unknown'
@@ -510,17 +500,17 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
         # What we're doing here now is first updating the allalbums & alltracks table to the most
         # current info, then moving the appropriate release into the album table and its associated
         # tracks into the tracks table
-            controlValueDict = {"ReleaseID" : release['ReleaseID']}
+            controlValueDict = {"ReleaseID": release['ReleaseID']}
 
-            newValueDict = {"ArtistID":         release['ArtistID'],
+            newValueDict = {"ArtistID": release['ArtistID'],
-                            "ArtistName":       release['ArtistName'],
+                            "ArtistName": release['ArtistName'],
-                            "AlbumTitle":       release['AlbumTitle'],
+                            "AlbumTitle": release['AlbumTitle'],
-                            "AlbumID":          release['AlbumID'],
+                            "AlbumID": release['AlbumID'],
-                            "AlbumASIN":        release['AlbumASIN'],
+                            "AlbumASIN": release['AlbumASIN'],
-                            "ReleaseDate":      release['ReleaseDate'],
+                            "ReleaseDate": release['ReleaseDate'],
-                            "Type":             release['Type'],
+                            "Type": release['Type'],
-                            "ReleaseCountry":   release['ReleaseCountry'],
+                            "ReleaseCountry": release['ReleaseCountry'],
-                            "ReleaseFormat":    release['ReleaseFormat']
+                            "ReleaseFormat": release['ReleaseFormat']
                         }
 
             myDB.upsert("allalbums", newValueDict, controlValueDict)
@@ -529,18 +519,18 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
 
                 cleanname = helpers.cleanName(release['ArtistName'] + ' ' + release['AlbumTitle'] + ' ' + track['title'])
 
-                controlValueDict = {"TrackID":      track['id'],
+                controlValueDict = {"TrackID": track['id'],
-                                    "ReleaseID":    release['ReleaseID']}
+                                    "ReleaseID": release['ReleaseID']}
 
-                newValueDict = {"ArtistID":         release['ArtistID'],
+                newValueDict = {"ArtistID": release['ArtistID'],
-                                "ArtistName":       release['ArtistName'],
+                                "ArtistName": release['ArtistName'],
-                                "AlbumTitle":       release['AlbumTitle'],
+                                "AlbumTitle": release['AlbumTitle'],
-                                "AlbumID":          release['AlbumID'],
+                                "AlbumID": release['AlbumID'],
-                                "AlbumASIN":        release['AlbumASIN'],
+                                "AlbumASIN": release['AlbumASIN'],
-                                "TrackTitle":       track['title'],
+                                "TrackTitle": track['title'],
-                                "TrackDuration":    track['duration'],
+                                "TrackDuration": track['duration'],
-                                "TrackNumber":      track['number'],
+                                "TrackNumber": track['number'],
-                                "CleanName":        cleanname
+                                "CleanName": cleanname
                             }
 
                 match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?', [cleanname]).fetchone()
@@ -558,20 +548,19 @@ def get_new_releases(rgid,includeExtras=False,forcefull=False):
 
                 myDB.upsert("alltracks", newValueDict, controlValueDict)
             num_new_releases = num_new_releases + 1
-            #print releasedata['title']
-            #print num_new_releases
             if album_checker:
                 logger.info('[%s] Existing release %s (%s) updated' % (release['ArtistName'], release['AlbumTitle'], rel_id_check))
             else:
                 logger.info('[%s] New release %s (%s) added' % (release['ArtistName'], release['AlbumTitle'], rel_id_check))
-    if force_repackage1 == 1:
+        if force_repackage1 == 1:
-        num_new_releases = -1
+            num_new_releases = -1
-        logger.info('[%s] Forcing repackage of %s, since dB releases have been removed' % (release['ArtistName'], release_title))
+            logger.info('[%s] Forcing repackage of %s, since dB releases have been removed' % (release['ArtistName'], release_title))
-    else:
+        else:
-        num_new_releases = num_new_releases
+            num_new_releases = num_new_releases
 
     return num_new_releases
 
+
 def getTracksFromRelease(release):
     totalTracks = 1
     tracks = []
@@ -582,16 +571,18 @@ def getTracksFromRelease(release):
             except:
                 track_title = unicode(track['recording']['title'])
             tracks.append({
-                    'number':        totalTracks,
+                    'number': totalTracks,
-                    'title':         track_title,
+                    'title': track_title,
-                    'id':            unicode(track['recording']['id']),
+                    'id': unicode(track['recording']['id']),
-                    'url':           u"http://musicbrainz.org/track/" + track['recording']['id'],
+                    'url': u"http://musicbrainz.org/track/" + track['recording']['id'],
-                    'duration':      int(track['length']) if 'length' in track else 0
+                    'duration': int(track['length']) if 'length' in track else 0
                     })
             totalTracks += 1
     return tracks
 
 # Used when there is a disambiguation
+
+
 def findArtistbyAlbum(name):
 
     myDB = db.DBConnection()
@@ -605,16 +596,16 @@ def findArtistbyAlbum(name):
     if not artist['AlbumTitle']:
         return False
 
-    term = '"'+artist['AlbumTitle']+'" AND artist:"'+name+'"'
+    term = '"' + artist['AlbumTitle'] + '" AND artist:"' + name + '"'
 
     results = None
 
     try:
-        results = musicbrainzngs.search_release_groups(term).get('release-group-list')
+        with mb_lock:
+            results = musicbrainzngs.search_release_groups(term).get('release-group-list')
     except musicbrainzngs.WebServiceError as e:
         logger.warn('Attempt to query MusicBrainz for %s failed (%s)' % (name, str(e)))
-        time.sleep(5)
+        mb_lock.snooze(5)
-
 
     if not results:
         return False
@@ -633,10 +624,9 @@ def findArtistbyAlbum(name):
         #artist_dict['url'] = u'http://musicbrainz.org/artist/' + newArtist['id']
         #artist_dict['score'] = int(releaseGroup['ext:score'])
 
-
-
     return artist_dict
 
+
 def findAlbumID(artist=None, album=None):
 
     results = None
@@ -645,20 +635,20 @@ def findAlbumID(artist=None, album=None):
     try:
         if album and artist:
             if any((c in chars) for c in album):
-                album = '"'+album+'"'
+                album = '"' + album + '"'
             if any((c in chars) for c in artist):
-                artist = '"'+artist+'"'
+                artist = '"' + artist + '"'
             criteria = {'release': album.lower()}
             criteria['artist'] = artist.lower()
         else:
             if any((c in chars) for c in album):
-                album = '"'+album+'"'
+                album = '"' + album + '"'
             criteria = {'release': album.lower()}
-
+        with mb_lock:
-        results = musicbrainzngs.search_release_groups(limit=1, **criteria).get('release-group-list')
+            results = musicbrainzngs.search_release_groups(limit=1, **criteria).get('release-group-list')
     except musicbrainzngs.WebServiceError as e:
         logger.warn('Attempt to query MusicBrainz for %s - %s failed (%s)' % (artist, album, str(e)))
-        time.sleep(5)
+        mb_lock.snooze(5)
 
     if not results:
         return False

headphones/music_encoder.py

@@ -14,36 +14,35 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import headphones
-import shutil
 import time
+import shutil
+import subprocess
+import headphones
 import multiprocessing
 
-import subprocess
 from headphones import logger
 from beets.mediafile import MediaFile
 
 # xld
-if headphones.ENCODER == 'xld':
+import getXldProfile
-    import getXldProfile
+
-    XLD = True
-else:
-    XLD = False
 
 def encode(albumPath):
+    use_xld = headphones.CONFIG.ENCODER == 'xld'
 
     # Return if xld details not found
-    if XLD:
+    if use_xld:
-        global xldProfile
+        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
-        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(headphones.XLDPROFILE)
         if not xldFormat:
             logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded', xldProfile)
             return None
+    else:
+        xldProfile = None
 
-    tempDirEncode=os.path.join(albumPath,"temp")
+    tempDirEncode = os.path.join(albumPath, "temp")
-    musicFiles=[]
+    musicFiles = []
-    musicFinalFiles=[]
+    musicFinalFiles = []
-    musicTempFiles=[]
+    musicTempFiles = []
     encoder = ""
 
     # Create temporary directory, but remove the old one first.
@@ -57,19 +56,19 @@ def encode(albumPath):
         logger.exception("Unable to create temporary directory")
         return None
 
-    for r,d,f in os.walk(albumPath):
+    for r, d, f in os.walk(albumPath):
         for music in f:
             if any(music.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
-                if not XLD:
+                if not use_xld:
-                    encoderFormat = headphones.ENCODEROUTPUTFORMAT.encode(headphones.SYS_ENCODING)
+                    encoderFormat = headphones.CONFIG.ENCODEROUTPUTFORMAT.encode(headphones.SYS_ENCODING)
                 else:
                     xldMusicFile = os.path.join(r, music)
                     xldInfoMusic = MediaFile(xldMusicFile)
                     encoderFormat = xldFormat
 
-                if (headphones.ENCODERLOSSLESS):
+                if headphones.CONFIG.ENCODERLOSSLESS:
                     ext = os.path.normpath(os.path.splitext(music)[1].lstrip(".")).lower()
-                    if not XLD and ext == 'flac' or XLD and (ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
+                    if not use_xld and ext == 'flac' or use_xld and (ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
                         musicFiles.append(os.path.join(r, music))
                         musicTemp = os.path.normpath(os.path.splitext(music)[0] + '.' + encoderFormat)
                         musicTempFiles.append(os.path.join(tempDirEncode, musicTemp))
@@ -80,29 +79,29 @@ def encode(albumPath):
                     musicTemp = os.path.normpath(os.path.splitext(music)[0] + '.' + encoderFormat)
                     musicTempFiles.append(os.path.join(tempDirEncode, musicTemp))
 
-    if headphones.ENCODER_PATH:
+    if headphones.CONFIG.ENCODER_PATH:
-        encoder = headphones.ENCODER_PATH.encode(headphones.SYS_ENCODING)
+        encoder = headphones.CONFIG.ENCODER_PATH.encode(headphones.SYS_ENCODING)
     else:
-        if XLD:
+        if use_xld:
             encoder = os.path.join('/Applications', 'xld')
-        elif headphones.ENCODER =='lame':
+        elif headphones.CONFIG.ENCODER == 'lame':
             if headphones.SYS_PLATFORM == "win32":
                 ## NEED THE DEFAULT LAME INSTALL ON WIN!
                 encoder = "C:/Program Files/lame/lame.exe"
             else:
-                encoder="lame"
+                encoder = "lame"
-        elif headphones.ENCODER =='ffmpeg':
+        elif headphones.CONFIG.ENCODER == 'ffmpeg':
             if headphones.SYS_PLATFORM == "win32":
                 encoder = "C:/Program Files/ffmpeg/bin/ffmpeg.exe"
             else:
-                encoder="ffmpeg"
+                encoder = "ffmpeg"
-        elif headphones.ENCODER == 'libav':
+        elif headphones.CONFIG.ENCODER == 'libav':
             if headphones.SYS_PLATFORM == "win32":
                 encoder = "C:/Program Files/libav/bin/avconv.exe"
             else:
-                encoder="avconv"
+                encoder = "avconv"
 
-    i=0
+    i = 0
     encoder_failed = False
     jobs = []
 
@@ -110,50 +109,50 @@ def encode(albumPath):
         infoMusic = MediaFile(music)
         encode = False
 
-        if XLD:
+        if use_xld:
             if xldBitrate and (infoMusic.bitrate / 1000 <= xldBitrate):
                 logger.info('%s has bitrate <= %skb, will not be re-encoded', music.decode(headphones.SYS_ENCODING, 'replace'), xldBitrate)
             else:
                 encode = True
-        elif headphones.ENCODER == 'lame':
+        elif headphones.CONFIG.ENCODER == 'lame':
             if not any(music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + x) for x in ["mp3", "wav"]):
                 logger.warn('Lame cannot encode %s format for %s, use ffmpeg', os.path.splitext(music)[1], music)
             else:
-                if (music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (int(infoMusic.bitrate / 1000) <= headphones.BITRATE)):
+                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
-                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.BITRATE)
+                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
                 else:
                     encode = True
         else:
-            if headphones.ENCODEROUTPUTFORMAT=='ogg':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                 if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.ogg'):
                     logger.warn('Cannot re-encode .ogg %s', music.decode(headphones.SYS_ENCODING, 'replace'))
                 else:
                     encode = True
-            elif (headphones.ENCODEROUTPUTFORMAT=='mp3' or headphones.ENCODEROUTPUTFORMAT=='m4a'):
+            elif headphones.CONFIG.ENCODEROUTPUTFORMAT == 'mp3' or headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
-                if (music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.'+headphones.ENCODEROUTPUTFORMAT) and (int(infoMusic.bitrate / 1000 ) <= headphones.BITRATE)):
+                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + headphones.CONFIG.ENCODEROUTPUTFORMAT) and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
-                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.BITRATE)
+                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
                 else:
                     encode = True
         # encode
         if encode:
-            job = (encoder, music, musicTempFiles[i], albumPath)
+            job = (encoder, music, musicTempFiles[i], albumPath, xldProfile)
             jobs.append(job)
         else:
             musicFiles[i] = None
             musicTempFiles[i] = None
 
-        i=i+1
+        i = i + 1
 
     # Encode music files
     if len(jobs) > 0:
         processes = 1
 
         # Use multicore if enabled
-        if headphones.ENCODER_MULTICORE:
+        if headphones.CONFIG.ENCODER_MULTICORE:
-            if headphones.ENCODER_MULTICORE_COUNT == 0:
+            if headphones.CONFIG.ENCODER_MULTICORE_COUNT == 0:
                 processes = multiprocessing.cpu_count()
             else:
-                processes = headphones.ENCODER_MULTICORE_COUNT
+                processes = headphones.CONFIG.ENCODER_MULTICORE_COUNT
 
             logger.debug("Multi-core encoding enabled, spawning %d processes",
                 processes)
@@ -194,14 +193,14 @@ def encode(albumPath):
         for dest in musicTempFiles:
             if os.path.exists(dest):
                 source = musicFiles[i]
-                if headphones.DELETE_LOSSLESS_FILES:
+                if headphones.CONFIG.DELETE_LOSSLESS_FILES:
                     os.remove(source)
                 check_dest = os.path.join(albumPath, os.path.split(dest)[1])
                 if os.path.exists(check_dest):
                     os.remove(check_dest)
                 try:
                     shutil.move(dest, albumPath)
-                except Exception, e:
+                except Exception as e:
                     logger.error('Could not move %s to %s: %s', dest, albumPath, e)
                     encoder_failed = True
                     break
@@ -212,11 +211,11 @@ def encode(albumPath):
 
     # Return with error if any encoding errors
     if encoder_failed:
-        logger.error("One or more files failed to encode. Ensure you have the latest version of %s installed.", headphones.ENCODER)
+        logger.error("One or more files failed to encode. Ensure you have the latest version of %s installed.", headphones.CONFIG.ENCODER)
         return None
 
     time.sleep(1)
-    for r,d,f in os.walk(albumPath):
+    for r, d, f in os.walk(albumPath):
         for music in f:
             if any(music.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
                 musicFinalFiles.append(os.path.join(r, music))
@@ -226,6 +225,7 @@ def encode(albumPath):
 
     return musicFinalFiles
 
+
 def command_map(args):
     """
     Wrapper for the '[multiprocessing.]map()' method, to unpack the arguments
@@ -239,11 +239,12 @@ def command_map(args):
     # Start encoding
     try:
         return command(*args)
-    except Exception as e:
+    except Exception:
         logger.exception("Encoder raised an exception.")
         return False
 
-def command(encoder, musicSource, musicDest, albumPath):
+
+def command(encoder, musicSource, musicDest, albumPath, xldProfile):
     """
     Encode a given music file with a certain encoder. Returns True on success,
     or False otherwise.
@@ -252,8 +253,7 @@ def command(encoder, musicSource, musicDest, albumPath):
     startMusicTime = time.time()
     cmd = []
 
-    # XLD
+    if xldProfile:
-    if XLD:
         xldDestDir = os.path.split(musicDest)[0]
         cmd = [encoder]
         cmd.extend([musicSource])
@@ -263,17 +263,17 @@ def command(encoder, musicSource, musicDest, albumPath):
         cmd.extend([xldDestDir])
 
     # Lame
-    elif headphones.ENCODER == 'lame':
+    elif headphones.CONFIG.ENCODER == 'lame':
         cmd = [encoder]
         opts = []
-        if not headphones.ADVANCEDENCODER:
+        if not headphones.CONFIG.ADVANCEDENCODER:
             opts.extend(['-h'])
-            if headphones.ENCODERVBRCBR=='cbr':
+            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['--resample', str(headphones.SAMPLINGFREQUENCY), '-b', str(headphones.BITRATE)])
+                opts.extend(['--resample', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-b', str(headphones.CONFIG.BITRATE)])
-            elif headphones.ENCODERVBRCBR=='vbr':
+            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
-                opts.extend(['-v', str(headphones.ENCODERQUALITY)])
+                opts.extend(['-v', str(headphones.CONFIG.ENCODERQUALITY)])
         else:
-            advanced = (headphones.ADVANCEDENCODER.split())
+            advanced = (headphones.CONFIG.ADVANCEDENCODER.split())
             for tok in advanced:
                 opts.extend([tok.encode(headphones.SYS_ENCODING)])
         opts.extend([musicSource])
@@ -281,42 +281,42 @@ def command(encoder, musicSource, musicDest, albumPath):
         cmd.extend(opts)
 
     # FFmpeg
-    elif headphones.ENCODER == 'ffmpeg':
+    elif headphones.CONFIG.ENCODER == 'ffmpeg':
         cmd = [encoder, '-i', musicSource]
         opts = []
-        if not headphones.ADVANCEDENCODER:
+        if not headphones.CONFIG.ADVANCEDENCODER:
-            if headphones.ENCODEROUTPUTFORMAT=='ogg':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                 opts.extend(['-acodec', 'libvorbis'])
-            if headphones.ENCODEROUTPUTFORMAT=='m4a':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
                 opts.extend(['-strict', 'experimental'])
-            if headphones.ENCODERVBRCBR=='cbr':
+            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['-ar', str(headphones.SAMPLINGFREQUENCY), '-ab', str(headphones.BITRATE) + 'k'])
+                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab', str(headphones.CONFIG.BITRATE) + 'k'])
-            elif headphones.ENCODERVBRCBR=='vbr':
+            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
-                opts.extend(['-aq', str(headphones.ENCODERQUALITY)])
+                opts.extend(['-aq', str(headphones.CONFIG.ENCODERQUALITY)])
             opts.extend(['-y', '-ac', '2', '-vn'])
         else:
-            advanced = (headphones.ADVANCEDENCODER.split())
+            advanced = (headphones.CONFIG.ADVANCEDENCODER.split())
             for tok in advanced:
                 opts.extend([tok.encode(headphones.SYS_ENCODING)])
         opts.extend([musicDest])
         cmd.extend(opts)
 
     # Libav
-    elif headphones.ENCODER == "libav":
+    elif headphones.CONFIG.ENCODER == "libav":
         cmd = [encoder, '-i', musicSource]
         opts = []
-        if not headphones.ADVANCEDENCODER:
+        if not headphones.CONFIG.ADVANCEDENCODER:
-            if headphones.ENCODEROUTPUTFORMAT=='ogg':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                 opts.extend(['-acodec', 'libvorbis'])
-            if headphones.ENCODEROUTPUTFORMAT=='m4a':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
                 opts.extend(['-strict', 'experimental'])
-            if headphones.ENCODERVBRCBR=='cbr':
+            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
-                opts.extend(['-ar', str(headphones.SAMPLINGFREQUENCY), '-ab', str(headphones.BITRATE) + 'k'])
+                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab', str(headphones.CONFIG.BITRATE) + 'k'])
-            elif headphones.ENCODERVBRCBR=='vbr':
+            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
-                opts.extend(['-aq', str(headphones.ENCODERQUALITY)])
+                opts.extend(['-aq', str(headphones.CONFIG.ENCODERQUALITY)])
             opts.extend(['-y', '-ac', '2', '-vn'])
         else:
-            advanced = (headphones.ADVANCEDENCODER.split())
+            advanced = (headphones.CONFIG.ADVANCEDENCODER.split())
             for tok in advanced:
                 opts.extend([tok.encode(headphones.SYS_ENCODING)])
         opts.extend([musicDest])
@@ -339,7 +339,7 @@ def command(encoder, musicSource, musicDest, albumPath):
     process = subprocess.Popen(cmd, startupinfo=startupinfo,
         stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
         stderr=subprocess.PIPE)
-    stdout, stderr = process.communicate(headphones.ENCODER)
+    stdout, stderr = process.communicate(headphones.CONFIG.ENCODER)
 
     # Error if return code not zero
     if process.returncode:
@@ -347,7 +347,7 @@ def command(encoder, musicSource, musicDest, albumPath):
         out = stdout if stdout else stderr
         out = out.decode(headphones.SYS_ENCODING, 'replace')
         outlast2lines = '\n'.join(out.splitlines()[-2:])
-        logger.error('%s error details: %s' % (headphones.ENCODER, outlast2lines))
+        logger.error('%s error details: %s' % (headphones.CONFIG.ENCODER, outlast2lines))
         out = out.rstrip("\n")
         logger.debug(out)
         encoded = False
@@ -357,10 +357,11 @@ def command(encoder, musicSource, musicDest, albumPath):
 
     return encoded
 
+
 def getTimeEncode(start):
-    seconds =int(time.time()-start)
+    seconds = int(time.time() - start)
     hours = seconds / 3600
-    seconds -= 3600*hours
+    seconds -= 3600 * hours
     minutes = seconds / 60
-    seconds -= 60*minutes
+    seconds -= 60 * minutes
-    return "%02d:%02d:%02d" % (hours, minutes, seconds)
+    return "%02d:%02d:%02d" % (hours, minutes, seconds)

headphones/notifiers.py

@@ -17,6 +17,7 @@ from headphones import logger, helpers, common, request
 
 from xml.dom import minidom
 from httplib import HTTPSConnection
+from urlparse import parse_qsl
 from urllib import urlencode
 from pynma import pynma
 
@@ -28,26 +29,20 @@ import headphones
 import os.path
 import subprocess
 import gntp.notifier
-import time
 import json
 
 import oauth2 as oauth
 import pythontwitter as twitter
 
-try:
-    from urlparse import parse_qsl
-except ImportError:
-    from cgi import parse_qsl
-
 class GROWL(object):
     """
-    Growl notifications, for OS X
+    Growl notifications, for OS X.
     """
 
     def __init__(self):
-        self.enabled = headphones.GROWL_ENABLED
+        self.enabled = headphones.CONFIG.GROWL_ENABLED
-        self.host = headphones.GROWL_HOST
+        self.host = headphones.CONFIG.GROWL_HOST
-        self.password = headphones.GROWL_PASSWORD
+        self.password = headphones.CONFIG.GROWL_PASSWORD
 
     def conf(self, options):
         return cherrypy.config['config'].get('Growl', options)
@@ -84,14 +79,18 @@ class GROWL(object):
         try:
             growl.register()
         except gntp.notifier.errors.NetworkError:
-            logger.info(u'Growl notification failed: network error')
+            logger.warning(u'Growl notification failed: network error')
             return
         except gntp.notifier.errors.AuthError:
-            logger.info(u'Growl notification failed: authentication error')
+            logger.warning(u'Growl notification failed: authentication error')
             return
 
+        # Fix message
+        message = message.encode(headphones.SYS_ENCODING, "replace")
+
         # Send it, including an image
-        image_file = os.path.join(str(headphones.PROG_DIR), 'data/images/headphoneslogo.png')
+        image_file = os.path.join(str(headphones.PROG_DIR),
+            "data/images/headphoneslogo.png")
 
         with open(image_file, 'rb') as f:
             image = f.read()
@@ -104,7 +103,7 @@ class GROWL(object):
                 icon=image
             )
         except gntp.notifier.errors.NetworkError:
-            logger.info(u'Growl notification failed: network error')
+            logger.warning(u'Growl notification failed: network error')
             return
 
         logger.info(u"Growl notifications sent.")
@@ -120,35 +119,36 @@ class GROWL(object):
 
         self.notify('ZOMG Lazors Pewpewpew!', 'Test Message')
 
+
 class PROWL(object):
     """
     Prowl notifications.
     """
 
     def __init__(self):
-        self.enabled = headphones.PROWL_ENABLED
+        self.enabled = headphones.CONFIG.PROWL_ENABLED
-        self.keys = headphones.PROWL_KEYS
+        self.keys = headphones.CONFIG.PROWL_KEYS
-        self.priority = headphones.PROWL_PRIORITY
+        self.priority = headphones.CONFIG.PROWL_PRIORITY
 
     def conf(self, options):
         return cherrypy.config['config'].get('Prowl', options)
 
     def notify(self, message, event):
-        if not headphones.PROWL_ENABLED:
+        if not headphones.CONFIG.PROWL_ENABLED:
             return
 
         http_handler = HTTPSConnection("api.prowlapp.com")
 
-        data = {'apikey': headphones.PROWL_KEYS,
+        data = {'apikey': headphones.CONFIG.PROWL_KEYS,
                 'application': 'Headphones',
                 'event': event,
                 'description': message.encode("utf-8"),
-                'priority': headphones.PROWL_PRIORITY }
+                'priority': headphones.CONFIG.PROWL_PRIORITY}
 
         http_handler.request("POST",
                                 "/publicapi/add",
-                                headers = {'Content-type': "application/x-www-form-urlencoded"},
+                                headers={'Content-type': "application/x-www-form-urlencoded"},
-                                body = urlencode(data))
+                                body=urlencode(data))
         response = http_handler.getresponse()
         request_status = response.status
 
@@ -173,6 +173,7 @@ class PROWL(object):
 
         self.notify('ZOMG Lazors Pewpewpew!', 'Test Message')
 
+
 class MPC(object):
     """
     MPC library update
@@ -182,8 +183,8 @@ class MPC(object):
 
         pass
 
-    def notify( self ):
+    def notify(self):
-        subprocess.call( ["mpc", "update"] )
+        subprocess.call(["mpc", "update"])
 
 
 class XBMC(object):
@@ -193,9 +194,9 @@ class XBMC(object):
 
     def __init__(self):
 
-        self.hosts = headphones.XBMC_HOST
+        self.hosts = headphones.CONFIG.XBMC_HOST
-        self.username = headphones.XBMC_USERNAME
+        self.username = headphones.CONFIG.XBMC_USERNAME
-        self.password = headphones.XBMC_PASSWORD
+        self.password = headphones.CONFIG.XBMC_PASSWORD
 
     def _sendhttp(self, host, command):
         url_command = urllib.urlencode(command)
@@ -226,7 +227,7 @@ class XBMC(object):
         hosts = [x.strip() for x in self.hosts.split(',')]
 
         for host in hosts:
-            logger.info('Sending library update command to XBMC @ '+host)
+            logger.info('Sending library update command to XBMC @ ' + host)
             request = self._sendjson(host, 'AudioLibrary.Scan')
 
             if not request:
@@ -241,17 +242,17 @@ class XBMC(object):
         time = "3000" # in ms
 
         for host in hosts:
-            logger.info('Sending notification command to XMBC @ '+host)
+            logger.info('Sending notification command to XMBC @ ' + host)
             try:
                 version = self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})['version']['major']
 
                 if version < 12: #Eden
                     notification = header + "," + message + "," + time + "," + albumartpath
-                    notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification('+notification+')'}
+                    notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification(' + notification + ')'}
                     request = self._sendhttp(host, notifycommand)
 
                 else: #Frodo
-                    params = {'title':header, 'message': message, 'displaytime': int(time), 'image': albumartpath}
+                    params = {'title': header, 'message': message, 'displaytime': int(time), 'image': albumartpath}
                     request = self._sendjson(host, 'GUI.ShowNotification', params)
 
                 if not request:
@@ -260,25 +261,26 @@ class XBMC(object):
             except Exception:
                 logger.error('Error sending notification request to XBMC')
 
+
 class LMS(object):
     """
     Class for updating a Logitech Media Server
     """
 
     def __init__(self):
-        self.hosts = headphones.LMS_HOST
+        self.hosts = headphones.CONFIG.LMS_HOST
 
     def _sendjson(self, host):
-        data = {'id': 1, 'method': 'slim.request', 'params': ["",["rescan"]]}
+        data = {'id': 1, 'method': 'slim.request', 'params': ["", ["rescan"]]}
         data = json.JSONEncoder().encode(data)
 
         content = {'Content-Type': 'application/json'}
 
-        req = urllib2.Request(host+'/jsonrpc.js', data, content)
+        req = urllib2.Request(host + '/jsonrpc.js', data, content)
 
         try:
             handle = urllib2.urlopen(req)
-        except Exception, e:
+        except Exception as e:
             logger.warn('Error opening LMS url: %s' % e)
             return
 
@@ -288,26 +290,27 @@ class LMS(object):
             return response['result']
         except:
             logger.warn('LMS returned error: %s' % response['error'])
-            return
+            return response['error']
 
     def update(self):
 
         hosts = [x.strip() for x in self.hosts.split(',')]
 
         for host in hosts:
-            logger.info('Sending library rescan command to LMS @ '+host)
+            logger.info('Sending library rescan command to LMS @ ' + host)
             request = self._sendjson(host)
 
-            if not request:
+            if request:
                 logger.warn('Error sending rescan request to LMS')
 
+
 class Plex(object):
     def __init__(self):
 
-        self.server_hosts = headphones.PLEX_SERVER_HOST
+        self.server_hosts = headphones.CONFIG.PLEX_SERVER_HOST
-        self.client_hosts = headphones.PLEX_CLIENT_HOST
+        self.client_hosts = headphones.CONFIG.PLEX_CLIENT_HOST
-        self.username = headphones.PLEX_USERNAME
+        self.username = headphones.CONFIG.PLEX_USERNAME
-        self.password = headphones.PLEX_PASSWORD
+        self.password = headphones.CONFIG.PLEX_PASSWORD
 
     def _sendhttp(self, host, command):
 
@@ -328,7 +331,7 @@ class Plex(object):
 
         try:
             handle = urllib2.urlopen(req)
-        except Exception, e:
+        except Exception as e:
             logger.warn('Error opening Plex url: %s' % e)
             return
 
@@ -344,7 +347,7 @@ class Plex(object):
         hosts = [x.strip() for x in self.server_hosts.split(',')]
 
         for host in hosts:
-            logger.info('Sending library update command to Plex Media Server@ '+host)
+            logger.info('Sending library update command to Plex Media Server@ ' + host)
             url = "%s/library/sections" % host
             try:
                 xml_sections = minidom.parse(urllib.urlopen(url))
@@ -362,7 +365,7 @@ class Plex(object):
                     url = "%s/library/sections/%s/refresh" % (host, s.getAttribute('key'))
                     try:
                         urllib.urlopen(url)
-                    except Exception, e:
+                    except Exception as e:
                         logger.warn("Error updating library section for Plex Media Server: %s" % e)
                         return False
 
@@ -375,10 +378,10 @@ class Plex(object):
         time = "3000" # in ms
 
         for host in hosts:
-            logger.info('Sending notification command to Plex Media Server @ '+host)
+            logger.info('Sending notification command to Plex Media Server @ ' + host)
             try:
                 notification = header + "," + message + "," + time + "," + albumartpath
-                notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification('+notification+')'}
+                notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification(' + notification + ')'}
                 request = self._sendhttp(host, notifycommand)
 
                 if not request:
@@ -387,11 +390,12 @@ class Plex(object):
             except:
                 logger.warn('Error sending notification request to Plex Media Server')
 
+
 class NMA(object):
     def notify(self, artist=None, album=None, snatched=None):
         title = 'Headphones'
-        api = headphones.NMA_APIKEY
+        api = headphones.CONFIG.NMA_APIKEY
-        nma_priority = headphones.NMA_PRIORITY
+        nma_priority = headphones.CONFIG.NMA_PRIORITY
 
         logger.debug(u"NMA title: " + title)
         logger.debug(u"NMA API: " + api)
@@ -413,7 +417,8 @@ class NMA(object):
         keys = api.split(',')
         p.addkey(keys)
 
-        if len(keys) > 1: batch = True
+        if len(keys) > 1:
+            batch = True
 
         response = p.push(title, event, message, priority=nma_priority, batch_mode=batch)
 
@@ -423,31 +428,31 @@ class NMA(object):
         else:
             return True
 
+
 class PUSHBULLET(object):
 
     def __init__(self):
-        self.apikey = headphones.PUSHBULLET_APIKEY
+        self.apikey = headphones.CONFIG.PUSHBULLET_APIKEY
-        self.deviceid = headphones.PUSHBULLET_DEVICEID
+        self.deviceid = headphones.CONFIG.PUSHBULLET_DEVICEID
 
     def conf(self, options):
         return cherrypy.config['config'].get('PUSHBULLET', options)
 
     def notify(self, message, event):
-        if not headphones.PUSHBULLET_ENABLED:
+        if not headphones.CONFIG.PUSHBULLET_ENABLED:
             return
 
         http_handler = HTTPSConnection("api.pushbullet.com")
 
-        data = {'device_iden': headphones.PUSHBULLET_DEVICEID,
+        data = {'type': "note",
-                'type': "note",
                 'title': "Headphones",
-                'body': message.encode("utf-8") }
+                'body': message.encode("utf-8")}
 
         http_handler.request("POST",
-                                "/api/pushes",
+                                "/v2/pushes",
-                                headers = {'Content-type': "application/x-www-form-urlencoded",
+                                headers={'Content-type': "application/json",
-                                            'Authorization' : 'Basic %s' % base64.b64encode(headphones.PUSHBULLET_APIKEY + ":") },
+                                            'Authorization': 'Basic %s' % base64.b64encode(headphones.CONFIG.PUSHBULLET_APIKEY + ":")},
-                                body = urlencode(data))
+                                body=json.dumps(data))
         response = http_handler.getresponse()
         request_status = response.status
         logger.debug(u"PushBullet response status: %r" % request_status)
@@ -476,13 +481,14 @@ class PUSHBULLET(object):
 
         self.notify('Main Screen Activate', 'Test Message')
 
+
 class PUSHALOT(object):
 
     def notify(self, message, event):
-        if not headphones.PUSHALOT_ENABLED:
+        if not headphones.CONFIG.PUSHALOT_ENABLED:
             return
 
-        pushalot_authorizationtoken = headphones.PUSHALOT_APIKEY
+        pushalot_authorizationtoken = headphones.CONFIG.PUSHALOT_APIKEY
 
         logger.debug(u"Pushalot event: " + event)
         logger.debug(u"Pushalot message: " + message)
@@ -492,12 +498,12 @@ class PUSHALOT(object):
 
         data = {'AuthorizationToken': pushalot_authorizationtoken,
                 'Title': event.encode('utf-8'),
-                'Body': message.encode("utf-8") }
+                'Body': message.encode("utf-8")}
 
         http_handler.request("POST",
                                 "/api/sendmessage",
-                                headers = {'Content-type': "application/x-www-form-urlencoded"},
+                                headers={'Content-type': "application/x-www-form-urlencoded"},
-                                body = urlencode(data))
+                                body=urlencode(data))
         response = http_handler.getresponse()
         request_status = response.status
 
@@ -515,6 +521,7 @@ class PUSHALOT(object):
                 logger.info(u"Pushalot notification failed.")
                 return False
 
+
 class Synoindex(object):
     def __init__(self, util_loc='/usr/syno/bin/synoindex'):
         self.util_loc = util_loc
@@ -551,15 +558,16 @@ class Synoindex(object):
             for path in path_list:
                 self.notify(path)
 
+
 class PUSHOVER(object):
 
     def __init__(self):
-        self.enabled = headphones.PUSHOVER_ENABLED
+        self.enabled = headphones.CONFIG.PUSHOVER_ENABLED
-        self.keys = headphones.PUSHOVER_KEYS
+        self.keys = headphones.CONFIG.PUSHOVER_KEYS
-        self.priority = headphones.PUSHOVER_PRIORITY
+        self.priority = headphones.CONFIG.PUSHOVER_PRIORITY
 
-        if headphones.PUSHOVER_APITOKEN:
+        if headphones.CONFIG.PUSHOVER_APITOKEN:
-            self.application_token = headphones.PUSHOVER_APITOKEN
+            self.application_token = headphones.CONFIG.PUSHOVER_APITOKEN
         else:
             self.application_token = "LdPCoy0dqC21ktsbEyAVCcwvQiVlsz"
 
@@ -567,21 +575,21 @@ class PUSHOVER(object):
         return cherrypy.config['config'].get('Pushover', options)
 
     def notify(self, message, event):
-        if not headphones.PUSHOVER_ENABLED:
+        if not headphones.CONFIG.PUSHOVER_ENABLED:
             return
 
         http_handler = HTTPSConnection("api.pushover.net")
 
         data = {'token': self.application_token,
-                'user': headphones.PUSHOVER_KEYS,
+                'user': headphones.CONFIG.PUSHOVER_KEYS,
                 'title': event,
                 'message': message.encode("utf-8"),
-                'priority': headphones.PUSHOVER_PRIORITY }
+                'priority': headphones.CONFIG.PUSHOVER_PRIORITY}
 
         http_handler.request("POST",
                                 "/1/messages.json",
-                                headers = {'Content-type': "application/x-www-form-urlencoded"},
+                                headers={'Content-type': "application/x-www-form-urlencoded"},
-                                body = urlencode(data))
+                                body=urlencode(data))
         response = http_handler.getresponse()
         request_status = response.status
         logger.debug(u"Pushover response status: %r" % request_status)
@@ -609,33 +617,33 @@ class PUSHOVER(object):
 
         self.notify('Main Screen Activate', 'Test Message')
 
+
 class TwitterNotifier(object):
 
     REQUEST_TOKEN_URL = 'https://api.twitter.com/oauth/request_token'
-    ACCESS_TOKEN_URL  = 'https://api.twitter.com/oauth/access_token'
+    ACCESS_TOKEN_URL = 'https://api.twitter.com/oauth/access_token'
     AUTHORIZATION_URL = 'https://api.twitter.com/oauth/authorize'
-    SIGNIN_URL        = 'https://api.twitter.com/oauth/authenticate'
+    SIGNIN_URL = 'https://api.twitter.com/oauth/authenticate'
 
     def __init__(self):
         self.consumer_key = "oYKnp2ddX5gbARjqX8ZAAg"
         self.consumer_secret = "A4Xkw9i5SjHbTk7XT8zzOPqivhj9MmRDR9Qn95YA9sk"
 
     def notify_snatch(self, title):
-        if headphones.TWITTER_ONSNATCH:
+        if headphones.CONFIG.TWITTER_ONSNATCH:
-            self._notifyTwitter(common.notifyStrings[common.NOTIFY_SNATCH]+': '+title+' at '+helpers.now())
+            self._notifyTwitter(common.notifyStrings[common.NOTIFY_SNATCH] + ': ' + title + ' at ' + helpers.now())
 
     def notify_download(self, title):
-        if headphones.TWITTER_ENABLED:
+        if headphones.CONFIG.TWITTER_ENABLED:
-            self._notifyTwitter(common.notifyStrings[common.NOTIFY_DOWNLOAD]+': '+title+' at '+helpers.now())
+            self._notifyTwitter(common.notifyStrings[common.NOTIFY_DOWNLOAD] + ': ' + title + ' at ' + helpers.now())
 
     def test_notify(self):
-        return self._notifyTwitter("This is a test notification from Headphones at "+helpers.now(), force=True)
+        return self._notifyTwitter("This is a test notification from Headphones at " + helpers.now(), force=True)
 
     def _get_authorization(self):
 
-        signature_method_hmac_sha1 = oauth.SignatureMethod_HMAC_SHA1() #@UnusedVariable
+        oauth_consumer = oauth.Consumer(key=self.consumer_key, secret=self.consumer_secret)
-        oauth_consumer             = oauth.Consumer(key=self.consumer_key, secret=self.consumer_secret)
+        oauth_client = oauth.Client(oauth_consumer)
-        oauth_client               = oauth.Client(oauth_consumer)
 
         logger.info('Requesting temp token from Twitter')
 
@@ -646,72 +654,71 @@ class TwitterNotifier(object):
         else:
             request_token = dict(parse_qsl(content))
 
-            headphones.TWITTER_USERNAME = request_token['oauth_token']
+            headphones.CONFIG.TWITTER_USERNAME = request_token['oauth_token']
-            headphones.TWITTER_PASSWORD = request_token['oauth_token_secret']
+            headphones.CONFIG.TWITTER_PASSWORD = request_token['oauth_token_secret']
 
-            return self.AUTHORIZATION_URL+"?oauth_token="+ request_token['oauth_token']
+            return self.AUTHORIZATION_URL + "?oauth_token=" + request_token['oauth_token']
 
     def _get_credentials(self, key):
         request_token = {}
 
-        request_token['oauth_token'] = headphones.TWITTER_USERNAME
+        request_token['oauth_token'] = headphones.CONFIG.TWITTER_USERNAME
-        request_token['oauth_token_secret'] = headphones.TWITTER_PASSWORD
+        request_token['oauth_token_secret'] = headphones.CONFIG.TWITTER_PASSWORD
         request_token['oauth_callback_confirmed'] = 'true'
 
         token = oauth.Token(request_token['oauth_token'], request_token['oauth_token_secret'])
         token.set_verifier(key)
 
-        logger.info('Generating and signing request for an access token using key '+key)
+        logger.info('Generating and signing request for an access token using key ' + key)
 
-        signature_method_hmac_sha1 = oauth.SignatureMethod_HMAC_SHA1() #@UnusedVariable
+        oauth_consumer = oauth.Consumer(key=self.consumer_key, secret=self.consumer_secret)
-        oauth_consumer             = oauth.Consumer(key=self.consumer_key, secret=self.consumer_secret)
+        logger.info('oauth_consumer: ' + str(oauth_consumer))
-        logger.info('oauth_consumer: '+str(oauth_consumer))
+        oauth_client = oauth.Client(oauth_consumer, token)
-        oauth_client  = oauth.Client(oauth_consumer, token)
+        logger.info('oauth_client: ' + str(oauth_client))
-        logger.info('oauth_client: '+str(oauth_client))
         resp, content = oauth_client.request(self.ACCESS_TOKEN_URL, method='POST', body='oauth_verifier=%s' % key)
-        logger.info('resp, content: '+str(resp)+','+str(content))
+        logger.info('resp, content: ' + str(resp) + ',' + str(content))
 
-        access_token  = dict(parse_qsl(content))
+        access_token = dict(parse_qsl(content))
-        logger.info('access_token: '+str(access_token))
+        logger.info('access_token: ' + str(access_token))
 
-        logger.info('resp[status] = '+str(resp['status']))
+        logger.info('resp[status] = ' + str(resp['status']))
         if resp['status'] != '200':
-            logger.info('The request for a token with did not succeed: '+str(resp['status']), logger.ERROR)
+            logger.info('The request for a token with did not succeed: ' + str(resp['status']), logger.ERROR)
             return False
         else:
             logger.info('Your Twitter Access Token key: %s' % access_token['oauth_token'])
             logger.info('Access Token secret: %s' % access_token['oauth_token_secret'])
-            headphones.TWITTER_USERNAME = access_token['oauth_token']
+            headphones.CONFIG.TWITTER_USERNAME = access_token['oauth_token']
-            headphones.TWITTER_PASSWORD = access_token['oauth_token_secret']
+            headphones.CONFIG.TWITTER_PASSWORD = access_token['oauth_token_secret']
             return True
 
-
     def _send_tweet(self, message=None):
 
-        username=self.consumer_key
+        username = self.consumer_key
-        password=self.consumer_secret
+        password = self.consumer_secret
-        access_token_key=headphones.TWITTER_USERNAME
+        access_token_key = headphones.CONFIG.TWITTER_USERNAME
-        access_token_secret=headphones.TWITTER_PASSWORD
+        access_token_secret = headphones.CONFIG.TWITTER_PASSWORD
 
-        logger.info(u"Sending tweet: "+message)
+        logger.info(u"Sending tweet: " + message)
 
         api = twitter.Api(username, password, access_token_key, access_token_secret)
 
         try:
             api.PostUpdate(message)
-        except Exception, e:
+        except Exception as e:
             logger.info(u"Error Sending Tweet: %s" % e)
             return False
 
         return True
 
     def _notifyTwitter(self, message='', force=False):
-        prefix = headphones.TWITTER_PREFIX
+        prefix = headphones.CONFIG.TWITTER_PREFIX
 
-        if not headphones.TWITTER_ENABLED and not force:
+        if not headphones.CONFIG.TWITTER_ENABLED and not force:
             return False
 
-        return self._send_tweet(prefix+": "+message)
+        return self._send_tweet(prefix + ": " + message)
+
 
 class OSX_NOTIFY(object):
 
@@ -723,6 +730,7 @@ class OSX_NOTIFY(object):
 
     def swizzle(self, cls, SEL, func):
         old_IMP = cls.instanceMethodForSelector_(SEL)
+
         def wrapper(self, *args, **kwargs):
             return func(self, old_IMP, *args, **kwargs)
         new_IMP = self.objc.selector(wrapper, selector=old_IMP.selector,
@@ -761,13 +769,14 @@ class OSX_NOTIFY(object):
             del pool
             return True
 
-        except Exception, e:
+        except Exception as e:
             logger.warn('Error sending OS X Notification: %s' % e)
             return False
 
     def swizzled_bundleIdentifier(self, original, swizzled):
         return 'ade.headphones.osxnotify'
 
+
 class BOXCAR(object):
 
     def __init__(self):
@@ -779,7 +788,7 @@ class BOXCAR(object):
                 message += '<br></br><a href="http://musicbrainz.org/release-group/%s">MusicBrainz</a>' % rgid
 
             data = urllib.urlencode({
-                'user_credentials': headphones.BOXCAR_TOKEN,
+                'user_credentials': headphones.CONFIG.BOXCAR_TOKEN,
                 'notification[title]': title.encode('utf-8'),
                 'notification[long_message]': message.encode('utf-8'),
                 'notification[sound]': "done"
@@ -794,12 +803,13 @@ class BOXCAR(object):
             logger.warn('Error sending Boxcar2 Notification: %s' % e)
             return False
 
+
 class SubSonicNotifier(object):
 
     def __init__(self):
-        self.host = headphones.SUBSONIC_HOST
+        self.host = headphones.CONFIG.SUBSONIC_HOST
-        self.username = headphones.SUBSONIC_USERNAME
+        self.username = headphones.CONFIG.SUBSONIC_USERNAME
-        self.password = headphones.SUBSONIC_PASSWORD
+        self.password = headphones.CONFIG.SUBSONIC_PASSWORD
 
     def notify(self, albumpaths):
         # Correct URL
@@ -811,4 +821,4 @@ class SubSonicNotifier(object):
 
         # Invoke request
         request.request_response(self.host + "musicFolderSettings.view?scanNow",
-            auth=(self.username, self.password))
+            auth=(self.username, self.password))

headphones/nzbget.py

@@ -19,37 +19,33 @@
 # along with Sick Beard.  If not, see <http://www.gnu.org/licenses/>.
 
 
-
 import httplib
-import datetime
 
 import headphones
 
 from base64 import standard_b64encode
 import xmlrpclib
 
-#from headphones.providers.generic import GenericProvider
-
 from headphones import logger
 
+
 def sendNZB(nzb):
 
     addToTop = False
     nzbgetXMLrpc = "%(username)s:%(password)s@%(host)s/xmlrpc"
 
-    if headphones.NZBGET_HOST == None:
+    if headphones.CONFIG.NZBGET_HOST is None:
         logger.error(u"No NZBget host found in configuration. Please configure it.")
         return False
 
-    if headphones.NZBGET_HOST.startswith('https://'):
+    if headphones.CONFIG.NZBGET_HOST.startswith('https://'):
         nzbgetXMLrpc = 'https://' + nzbgetXMLrpc
-        headphones.NZBGET_HOST.replace('https://','',1)
+        headphones.CONFIG.NZBGET_HOST.replace('https://', '', 1)
     else:
         nzbgetXMLrpc = 'http://' + nzbgetXMLrpc
-        headphones.NZBGET_HOST.replace('http://','',1)
+        headphones.CONFIG.NZBGET_HOST.replace('http://', '', 1)
 
-
+    url = nzbgetXMLrpc % {"host": headphones.CONFIG.NZBGET_HOST, "username": headphones.CONFIG.NZBGET_USERNAME, "password": headphones.CONFIG.NZBGET_PASSWORD}
-    url = nzbgetXMLrpc % {"host": headphones.NZBGET_HOST, "username": headphones.NZBGET_USERNAME, "password": headphones.NZBGET_PASSWORD}
 
     nzbGetRPC = xmlrpclib.ServerProxy(url)
     try:
@@ -63,7 +59,7 @@ def sendNZB(nzb):
         return False
 
     except xmlrpclib.ProtocolError, e:
-        if (e.errmsg == "Unauthorized"):
+        if e.errmsg == "Unauthorized":
             logger.error(u"NZBget password is incorrect.")
         else:
             logger.error(u"Protocol Error: " + e.errmsg)
@@ -86,35 +82,37 @@ def sendNZB(nzb):
         nzbget_version = int(nzbget_version_str[:nzbget_version_str.find(".")])
         if nzbget_version == 0:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, addToTop, nzbcontent64)
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, addToTop, nzbcontent64)
             else:
-                if nzb.resultType == "nzb":
+                # from headphones.common.providers.generic import GenericProvider
-                    genProvider = GenericProvider("")
+                # if nzb.resultType == "nzb":
-                    data = genProvider.getURL(nzb.url)
+                #     genProvider = GenericProvider("")
-                    if (data == None):
+                #     data = genProvider.getURL(nzb.url)
-                        return False
+                #     if (data is None):
-                    nzbcontent64 = standard_b64encode(data)
+                #         return False
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, addToTop, nzbcontent64)
+                #     nzbcontent64 = standard_b64encode(data)
+                # nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, addToTop, nzbcontent64)
+                return False
         elif nzbget_version == 12:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
                                                  nzbcontent64, False, dupekey, dupescore, "score")
             else:
-                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
                                                     nzb.url, False, dupekey, dupescore, "score")
         # v13+ has a new combined append method that accepts both (url and content)
         # also the return value has changed from boolean to integer
         # (Positive number representing NZBID of the queue item. 0 and negative numbers represent error codes.)
         elif nzbget_version >= 13:
             nzbget_result = True if nzbGetRPC.append(nzb.name + ".nzb", nzbcontent64 if nzbcontent64 is not None else nzb.url,
-                                                     headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False, False, dupekey, dupescore,
+                                                     headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False, False, dupekey, dupescore,
                                                      "score") > 0 else False
         else:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
                                                  nzbcontent64)
             else:
-                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, headphones.CONFIG.NZBGET_PRIORITY, False,
                                                     nzb.url)
 
         if nzbget_result:

headphones/request.py

@@ -21,13 +21,26 @@ from bs4 import BeautifulSoup
 import requests
 import feedparser
 import headphones
+import headphones.lock
+import collections
+
+# Dictionary with last request times, for rate limiting.
+last_requests = collections.defaultdict(int)
+fake_lock = headphones.lock.FakeLock()
+
 
 def request_response(url, method="get", auto_raise=True,
-    whitelist_status_code=None, **kwargs):
+    whitelist_status_code=None, lock=fake_lock, **kwargs):
     """
     Convenient wrapper for `requests.get', which will capture the exceptions and
     log them. On success, the Response object is returned. In case of a
     exception, None is returned.
+
+    Additionally, there is support for rate limiting. To use this feature,
+    supply a tuple of (lock, request_limit). The lock is used to make sure no
+    other request with the same lock is executed. The request limit is the
+    minimal time between two requests (and so 1/request_limit is the number of
+    requests per seconds).
     """
 
     # Convert whitelist_status_code to a list if needed
@@ -36,16 +49,17 @@ def request_response(url, method="get", auto_raise=True,
 
     # Disable verification of SSL certificates if requested. Note: this could
     # pose a security issue!
-    kwargs["verify"] = headphones.VERIFY_SSL_CERT
+    kwargs["verify"] = bool(headphones.CONFIG.VERIFY_SSL_CERT)
 
     # Map method to the request.XXX method. This is a simple hack, but it allows
     # requests to apply more magic per method. See lib/requests/api.py.
     request_method = getattr(requests, method.lower())
 
     try:
-        # Request the URL
+        # Request URL and wait for response
-        logger.debug("Requesting URL via %s method: %s", method.upper(), url)
+        with lock:
-        response = request_method(url, **kwargs)
+            logger.debug("Requesting URL via %s method: %s", method.upper(), url)
+            response = request_method(url, **kwargs)
 
         # If status code != OK, then raise exception, except if the status code
         # is white listed.
@@ -54,19 +68,25 @@ def request_response(url, method="get", auto_raise=True,
                 try:
                     response.raise_for_status()
                 except:
-                    logger.debug("Response status code %d is not white " \
+                    logger.debug(
-                         "listed, raised exception", response.status_code)
+                        "Response status code %d is not white "
+                        "listed, raised exception", response.status_code)
                     raise
         elif auto_raise:
             response.raise_for_status()
 
         return response
+    except requests.exceptions.SSLError:
+        logger.error("Unable to connect to remote host because of a SSL " \
+            "error. It's likely the remote certificate is untrusted by your " \
+            "system. This check can be disabled (advanced users only).")
     except requests.ConnectionError:
-        logger.error("Unable to connect to remote host. Check if the remote " \
+        logger.error(
+            "Unable to connect to remote host. Check if the remote "
             "host is up and running.")
     except requests.Timeout:
-        logger.error("Request timed out. The remote host did not respeond " \
+        logger.error(
-            "timely.")
+            "Request timed out. The remote host did not respond timely.")
     except requests.HTTPError as e:
         if e.response is not None:
             if e.response.status_code >= 500:
@@ -88,6 +108,7 @@ def request_response(url, method="get", auto_raise=True,
     except requests.RequestException as e:
         logger.error("Request raised exception: %s", e)
 
+
 def request_soup(url, **kwargs):
     """
     Wrapper for `request_response', which will return a BeatifulSoup object if
@@ -100,6 +121,7 @@ def request_soup(url, **kwargs):
     if response is not None:
         return BeautifulSoup(response.content, parser)
 
+
 def request_minidom(url, **kwargs):
     """
     Wrapper for `request_response', which will return a Minidom object if no
@@ -111,6 +133,7 @@ def request_minidom(url, **kwargs):
     if response is not None:
         return minidom.parseString(response.content)
 
+
 def request_json(url, **kwargs):
     """
     Wrapper for `request_response', which will decode the response as JSON
@@ -138,6 +161,7 @@ def request_json(url, **kwargs):
             if headphones.VERBOSE:
                 server_message(response)
 
+
 def request_content(url, **kwargs):
     """
     Wrapper for `request_response', which will return the raw content.
@@ -148,6 +172,7 @@ def request_content(url, **kwargs):
     if response is not None:
         return response.content
 
+
 def request_feed(url, **kwargs):
     """
     Wrapper for `request_response', which will return a feed object.
@@ -158,6 +183,7 @@ def request_feed(url, **kwargs):
     if response is not None:
         return feedparser.parse(response.content)
 
+
 def server_message(response):
     """
     Extract server message from response and log in to logger with DEBUG level.
@@ -198,4 +224,4 @@ def server_message(response):
         if len(message) > 150:
             message = message[:150] + "..."
 
-        logger.debug("Server responded with message: %s", message)
+        logger.debug("Server responded with message: %s", message)

headphones/sab.py

@@ -19,7 +19,6 @@
 
 import MultipartPostHandler
 import headphones
-import datetime
 import cookielib
 import urllib2
 import httplib
@@ -28,20 +27,21 @@ import ast
 
 from headphones.common import USER_AGENT
 from headphones import logger
-from headphones import notifiers, helpers
+from headphones import helpers
+
 
 def sendNZB(nzb):
 
     params = {}
 
-    if headphones.SAB_USERNAME:
+    if headphones.CONFIG.SAB_USERNAME:
-        params['ma_username'] = headphones.SAB_USERNAME
+        params['ma_username'] = headphones.CONFIG.SAB_USERNAME
-    if headphones.SAB_PASSWORD:
+    if headphones.CONFIG.SAB_PASSWORD:
-        params['ma_password'] = headphones.SAB_PASSWORD
+        params['ma_password'] = headphones.CONFIG.SAB_PASSWORD
-    if headphones.SAB_APIKEY:
+    if headphones.CONFIG.SAB_APIKEY:
-        params['apikey'] = headphones.SAB_APIKEY
+        params['apikey'] = headphones.CONFIG.SAB_APIKEY
-    if headphones.SAB_CATEGORY:
+    if headphones.CONFIG.SAB_CATEGORY:
-        params['cat'] = headphones.SAB_CATEGORY
+        params['cat'] = headphones.CONFIG.SAB_CATEGORY
 
     # if it's a normal result we just pass SAB the URL
     if nzb.resultType == "nzb":
@@ -49,7 +49,7 @@ def sendNZB(nzb):
         if nzb.provider.getID() == 'newzbin':
             id = nzb.provider.getIDFromURL(nzb.url)
             if not id:
-                logger.info("Unable to send NZB to sab, can't find ID in URL "+str(nzb.url))
+                logger.info("Unable to send NZB to sab, can't find ID in URL " + str(nzb.url))
                 return False
             params['mode'] = 'addid'
             params['name'] = id
@@ -62,15 +62,15 @@ def sendNZB(nzb):
         # Sanitize the file a bit, since we can only use ascii chars with MultiPartPostHandler
         nzbdata = helpers.latinToAscii(nzb.extraInfo[0])
         params['mode'] = 'addfile'
-        multiPartParams = {"nzbfile": (helpers.latinToAscii(nzb.name)+".nzb", nzbdata)}
+        multiPartParams = {"nzbfile": (helpers.latinToAscii(nzb.name) + ".nzb", nzbdata)}
 
-    if not headphones.SAB_HOST.startswith('http'):
+    if not headphones.CONFIG.SAB_HOST.startswith('http'):
-        headphones.SAB_HOST = 'http://' + headphones.SAB_HOST
+        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
 
-    if headphones.SAB_HOST.endswith('/'):
+    if headphones.CONFIG.SAB_HOST.endswith('/'):
-        headphones.SAB_HOST = headphones.SAB_HOST[0:len(headphones.SAB_HOST)-1]
+        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[0:len(headphones.CONFIG.SAB_HOST) - 1]
 
-    url = headphones.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
+    url = headphones.CONFIG.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
 
     try:
 
@@ -87,25 +87,25 @@ def sendNZB(nzb):
 
             f = opener.open(req)
 
-    except (EOFError, IOError), e:
+    except (EOFError, IOError) as e:
         logger.error(u"Unable to connect to SAB with URL: %s" % url)
         return False
 
-    except httplib.InvalidURL, e:
+    except httplib.InvalidURL as e:
-        logger.error(u"Invalid SAB host, check your config. Current host: %s" % headphones.SAB_HOST)
+        logger.error(u"Invalid SAB host, check your config. Current host: %s" % headphones.CONFIG.SAB_HOST)
         return False
-        
+
-    except Exception, e:
+    except Exception as e:
         logger.error(u"Error: " + str(e))
         return False
-        
+
-    if f == None:
+    if f is None:
         logger.info(u"No data returned from SABnzbd, NZB not sent")
         return False
 
     try:
         result = f.readlines()
-    except Exception, e:
+    except Exception as e:
         logger.info(u"Error trying to get result from SAB, NZB not sent: ")
         return False
 
@@ -126,37 +126,38 @@ def sendNZB(nzb):
     else:
         logger.info(u"Unknown failure sending NZB to sab. Return text is: " + sabText)
         return False
-        
+
+
 def checkConfig():
 
-    params = { 'mode' : 'get_config', 
+    params = {'mode': 'get_config',
-               'section' : 'misc' 
+               'section': 'misc'
                }
 
-    if headphones.SAB_USERNAME:
+    if headphones.CONFIG.SAB_USERNAME:
-        params['ma_username'] = headphones.SAB_USERNAME
+        params['ma_username'] = headphones.CONFIG.SAB_USERNAME
-    if headphones.SAB_PASSWORD:
+    if headphones.CONFIG.SAB_PASSWORD:
-        params['ma_password'] = headphones.SAB_PASSWORD
+        params['ma_password'] = headphones.CONFIG.SAB_PASSWORD
-    if headphones.SAB_APIKEY:
+    if headphones.CONFIG.SAB_APIKEY:
-        params['apikey'] = headphones.SAB_APIKEY
+        params['apikey'] = headphones.CONFIG.SAB_APIKEY
 
-    if not headphones.SAB_HOST.startswith('http'):
+    if not headphones.CONFIG.SAB_HOST.startswith('http'):
-        headphones.SAB_HOST = 'http://' + headphones.SAB_HOST
+        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
+
+    if headphones.CONFIG.SAB_HOST.endswith('/'):
+        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[0:len(headphones.CONFIG.SAB_HOST) - 1]
+
+    url = headphones.CONFIG.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
 
-    if headphones.SAB_HOST.endswith('/'):
-        headphones.SAB_HOST = headphones.SAB_HOST[0:len(headphones.SAB_HOST)-1]
-    
-    url = headphones.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
-    
     try:
         f = urllib.urlopen(url).read()
-    except Exception, e:
+    except Exception:
         logger.warn("Unable to read SABnzbd config file - cannot determine renaming options (might affect auto & forced post processing)")
         return (0, 0)
-        
+
     config_options = ast.literal_eval(f)
-    
+
     replace_spaces = config_options['misc']['replace_spaces']
     replace_dots = config_options['misc']['replace_dots']
-    
+
     return (replace_spaces, replace_dots)

headphones/searcher_rutracker.py

@@ -4,8 +4,6 @@
 # Headphones rutracker.org search
 # Functions called from searcher.py
 
-from headphones import logger, db, utorrent
-
 from bencode import bencode as bencode, bdecode
 from urlparse import urlparse
 from bs4 import BeautifulSoup
@@ -20,6 +18,9 @@ import urllib
 import re
 import os
 
+from headphones import db, logger
+
+
 class Rutracker():
 
     logged_in = False
@@ -47,9 +48,9 @@ class Rutracker():
         #if self.login_counter > 1:
         #    return False
 
-        params = urllib.urlencode({"login_username" : login,
+        params = urllib.urlencode({"login_username": login,
-                                   "login_password" : password,
+                                   "login_password": password,
-                                   "login" : "Вход"})
+                                   "login": "Вход"})
 
         try:
             self.opener.open("http://login.rutracker.org/forum/login.php", params)
@@ -114,26 +115,26 @@ class Rutracker():
             #logger.debug (soup.prettify())
 
             # Title
-            for link in soup.find_all('a', attrs={'class' : 'med tLink hl-tags bold'}):
+            for link in soup.find_all('a', attrs={'class': 'med tLink hl-tags bold'}):
                 title = link.get_text()
                 titles.append(title)
 
             # Download URL
-            for link in soup.find_all('a', attrs={'class' : 'small tr-dl dl-stub'}):
+            for link in soup.find_all('a', attrs={'class': 'small tr-dl dl-stub'}):
                 url = link.get('href')
                 urls.append(url)
 
             # Seeders
-            for link in soup.find_all('b', attrs={'class' : 'seedmed'}):
+            for link in soup.find_all('b', attrs={'class': 'seedmed'}):
                 seeder = link.get_text()
                 seeders.append(seeder)
 
             # Size
-            for link in soup.find_all('td', attrs={'class' : 'row4 small nowrap tor-size'}):
+            for link in soup.find_all('td', attrs={'class': 'row4 small nowrap tor-size'}):
                 size = link.u.string
                 sizes.append(size)
 
-        except :
+        except:
             pass
 
         # Combine lists
@@ -196,8 +197,8 @@ class Rutracker():
                         if torrent:
                             decoded = bdecode(torrent)
                             metainfo = decoded['info']
-                        page.close ()
+                        page.close()
-                    except Exception, e:
+                    except Exception as e:
                         logger.error('Error getting torrent: %s' % e)
                         return False
 
@@ -215,9 +216,9 @@ class Rutracker():
                                     cuecount += 1
 
                     title = returntitle.lower()
-                    logger.debug ('torrent title: %s' % title)
+                    logger.debug('torrent title: %s' % title)
-                    logger.debug ('headphones trackcount: %s' % hptrackcount)
+                    logger.debug('headphones trackcount: %s' % hptrackcount)
-                    logger.debug ('rutracker trackcount: %s' % trackcount)
+                    logger.debug('rutracker trackcount: %s' % trackcount)
 
                     # If torrent track count less than headphones track count, and there's a cue, then attempt to get track count from log(s)
                     # This is for the case where we have a single .flac/.wav which can be split by cue
@@ -245,7 +246,7 @@ class Rutracker():
 
                     if totallogcount > 0:
                         trackcount = totallogcount
-                        logger.debug ('rutracker logtrackcount: %s' % totallogcount)
+                        logger.debug('rutracker logtrackcount: %s' % totallogcount)
 
                     # If torrent track count = hp track count then return torrent,
                     # if greater, check for deluxe/special/foreign editions
@@ -294,7 +295,7 @@ class Rutracker():
             os.umask(prev)
 
             # Add file to utorrent
-            if headphones.TORRENT_DOWNLOADER == 2:
+            if headphones.CONFIG.TORRENT_DOWNLOADER == 2:
                 self.utorrent_add_file(download_path)
 
         except Exception as e:
@@ -306,7 +307,7 @@ class Rutracker():
     #TODO get this working in utorrent.py
     def utorrent_add_file(self, filename):
 
-        host = headphones.UTORRENT_HOST
+        host = headphones.CONFIG.UTORRENT_HOST
         if not host.startswith('http'):
             host = 'http://' + host
         if host.endswith('/'):
@@ -315,8 +316,8 @@ class Rutracker():
             host = host[:-4]
 
         base_url = host
-        username = headphones.UTORRENT_USERNAME
+        username = headphones.CONFIG.UTORRENT_USERNAME
-        password = headphones.UTORRENT_PASSWORD
+        password = headphones.CONFIG.UTORRENT_PASSWORD
 
         session = requests.Session()
         url = base_url + '/gui/'
@@ -346,4 +347,3 @@ class Rutracker():
             except Exception:
                 logger.exception('Error adding file to utorrent')
                 return
-

headphones/torrentfinished.py

@@ -13,19 +13,21 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
+from headphones import db, utorrent, transmission, logger
+
 import threading
 import headphones
-from headphones import db, utorrent, transmission, logger
 
 postprocessor_lock = threading.Lock()
 
-# Remove Torrent + data if Post Processed and finished Seeding
 def checkTorrentFinished():
+    """
+    Remove Torrent + data if Post Processed and finished Seeding
+    """
 
     logger.info("Checking if any torrents have finished seeding and can be removed")
 
     with postprocessor_lock:
-
         myDB = db.DBConnection()
         results = myDB.select('SELECT * from snatched WHERE Status="Seed_Processed"')
 
@@ -33,10 +35,12 @@ def checkTorrentFinished():
             hash = album['FolderName']
             albumid = album['AlbumID']
             torrent_removed = False
-            if headphones.TORRENT_DOWNLOADER == 1:
+            if headphones.CONFIG.TORRENT_DOWNLOADER == 1:
                 torrent_removed = transmission.removeTorrent(hash, True)
             else:
                 torrent_removed = utorrent.removeTorrent(hash, True)
 
             if torrent_removed:
                 myDB.action('DELETE from snatched WHERE status = "Seed_Processed" and AlbumID=?', [albumid])
+
+    logger.info("Checking finished torrents completed")

headphones/transmission.py

@@ -13,12 +13,12 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger, notifiers, request
+from headphones import logger, request
 
-import re
 import time
 import json
 import base64
+import urlparse
 import headphones
 
 # This is just a simple script to send torrents to transmission. The
@@ -27,30 +27,28 @@ import headphones
 # TODO: Store the session id so we don't need to make 2 calls
 #       Store torrent id so we can check up on it
 
+
 def addTorrent(link):
     method = 'torrent-add'
 
     if link.endswith('.torrent'):
         with open(link, 'rb') as f:
             metainfo = str(base64.b64encode(f.read()))
-        arguments = {'metainfo': metainfo, 'download-dir':headphones.DOWNLOAD_TORRENT_DIR}
+        arguments = {'metainfo': metainfo, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}
     else:
-        arguments = {'filename': link, 'download-dir': headphones.DOWNLOAD_TORRENT_DIR}
+        arguments = {'filename': link, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}
 
-    response = torrentAction(method,arguments)
+    response = torrentAction(method, arguments)
 
     if not response:
         return False
 
     if response['result'] == 'success':
         if 'torrent-added' in response['arguments']:
-            name = response['arguments']['torrent-added']['name']
             retid = response['arguments']['torrent-added']['hashString']
         elif 'torrent-duplicate' in response['arguments']:
-            name = response['arguments']['torrent-duplicate']['name']
             retid = response['arguments']['torrent-duplicate']['hashString']
         else:
-            name = link
             retid = False
 
         logger.info(u"Torrent sent to Transmission successfully")
@@ -60,9 +58,10 @@ def addTorrent(link):
         logger.info('Transmission returned status %s' % response['result'])
         return False
 
+
 def getTorrentFolder(torrentid):
     method = 'torrent-get'
-    arguments = { 'ids': torrentid, 'fields': ['name','percentDone']}
+    arguments = {'ids': torrentid, 'fields': ['name', 'percentDone']}
 
     response = torrentAction(method, arguments)
     percentdone = response['arguments']['torrents'][0]['percentDone']
@@ -70,8 +69,8 @@ def getTorrentFolder(torrentid):
 
     tries = 1
 
-    while percentdone == 0  and tries <10:
+    while percentdone == 0 and tries < 10:
-        tries+=1
+        tries += 1
         time.sleep(5)
         response = torrentAction(method, arguments)
         percentdone = response['arguments']['torrents'][0]['percentDone']
@@ -80,6 +79,7 @@ def getTorrentFolder(torrentid):
 
     return torrent_folder_name
 
+
 def setSeedRatio(torrentid, ratio):
     method = 'torrent-set'
     if ratio != 0:
@@ -91,10 +91,11 @@ def setSeedRatio(torrentid, ratio):
     if not response:
         return False
 
-def removeTorrent(torrentid, remove_data = False):
+
+def removeTorrent(torrentid, remove_data=False):
 
     method = 'torrent-get'
-    arguments = { 'ids': torrentid, 'fields': ['isFinished', 'name']}
+    arguments = {'ids': torrentid, 'fields': ['isFinished', 'name']}
 
     response = torrentAction(method, arguments)
     if not response:
@@ -120,12 +121,12 @@ def removeTorrent(torrentid, remove_data = False):
 
     return False
 
+
 def torrentAction(method, arguments):
 
-    host = headphones.TRANSMISSION_HOST
+    host = headphones.CONFIG.TRANSMISSION_HOST
-    username = headphones.TRANSMISSION_USERNAME
+    username = headphones.CONFIG.TRANSMISSION_USERNAME
-    password = headphones.TRANSMISSION_PASSWORD
+    password = headphones.CONFIG.TRANSMISSION_PASSWORD
-    sessionid = None
 
     if not host.startswith('http'):
         host = 'http://' + host
@@ -133,28 +134,22 @@ def torrentAction(method, arguments):
     if host.endswith('/'):
         host = host[:-1]
 
-    # Either the host ends with a port, or some directory, or rpc
+    # Fix the URL. We assume that the user does not point to the RPC endpoint,
-    # If it ends with /rpc we don't have to do anything
+    # so add it if it is missing.
-    # If it ends with a port we add /transmission/rpc
+    parts = list(urlparse.urlparse(host))
-    # anything else we just add rpc
+
-    if not host.endswith('/rpc'):
+    if not parts[0] in ("http", "https"):
-        # Check if it ends in a port number
+        parts[0] = "http"
-        i = host.rfind(':')
+
-        if i >= 0:
+    if not parts[2].endswith("/rpc"):
-            possible_port = host[i+1:]
+        parts[2] += "/transmission/rpc"
-            try:
+
-                port = int(possible_port)
+    host = urlparse.urlunparse(parts)
-                host = host + "/transmission/rpc"
-            except ValueError:
-                host = host + "/rpc"
-        else:
-            logger.error('Transmission port missing')
-            return
 
     # Retrieve session id
     auth = (username, password) if username and password else None
-
+    response = request.request_response(host, auth=auth,
-    response = request.request_response(host, auth=auth, whitelist_status_code=[401, 409])
+        whitelist_status_code=[401, 409])
 
     if response is None:
         logger.error("Error gettings Transmission session ID")
@@ -163,23 +158,27 @@ def torrentAction(method, arguments):
     # Parse response
     if response.status_code == 401:
         if auth:
-            logger.error("Username and/or password not accepted by Transmission")
+            logger.error("Username and/or password not accepted by " \
+                "Transmission")
         else:
             logger.error("Transmission authorization required")
 
         return
     elif response.status_code == 409:
-        sessionid = response.headers['x-transmission-session-id']
+        session_id = response.headers['x-transmission-session-id']
 
-    if not sessionid:
+        if not session_id:
-        logger.error("Error getting Session ID from Transmission")
+            logger.error("Expected a Session ID from Transmission")
-        return
+            return
 
     # Prepare next request
-    headers = { 'x-transmission-session-id': sessionid }
+    headers = {'x-transmission-session-id': session_id}
-    data = { 'method': method, 'arguments': arguments }
+    data = {'method': method, 'arguments': arguments}
 
-    response = request.request_json(host, method="post", data=json.dumps(data), headers=headers, auth=auth)
+    response = request.request_json(host, method="POST", data=json.dumps(data),
+        headers=headers, auth=auth)
+
+    print response
 
     if not response:
         logger.error("Error sending torrent to Transmission")

headphones/updater.py

@@ -13,19 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import headphones
-
 from headphones import logger, db, importer
 
+
 def dbUpdate(forcefull=False):
 
     myDB = db.DBConnection()
 
-    activeartists = myDB.select('SELECT ArtistID, ArtistName from artists WHERE Status="Active" or Status="Loading" order by LastUpdated ASC')
+    active_artists = myDB.select('SELECT ArtistID, ArtistName from artists WHERE Status="Active" or Status="Loading" order by LastUpdated ASC')
-    logger.info('Starting update for %i active artists' % len(activeartists))
+    logger.info('Starting update for %i active artists', len(active_artists))
-
-    for artist in activeartists:
 
+    for artist in active_artists:
         artistid = artist[0]
         importer.addArtisttoDB(artistid=artistid, extrasonly=False, forcefull=forcefull)
 

headphones/utorrent.py

@@ -13,22 +13,29 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import urllib, urllib2, urlparse, cookielib
+import urllib
-import json, re, os, time
+import urllib2
+import urlparse
+import cookielib
+import json
+import re
+import os
+import time
 
 import headphones
 
 from headphones import logger
 from collections import namedtuple
 
+
 class utorrentclient(object):
 
     TOKEN_REGEX = "<div id='token' style='display:none;'>([^<>]+)</div>"
     UTSetting = namedtuple("UTSetting", ["name", "int", "str", "access"])
 
-    def __init__(self, base_url = None, username = None, password = None,):
+    def __init__(self, base_url=None, username=None, password=None,):
 
-        host = headphones.UTORRENT_HOST
+        host = headphones.CONFIG.UTORRENT_HOST
         if not host.startswith('http'):
             host = 'http://' + host
 
@@ -39,8 +46,8 @@ class utorrentclient(object):
             host = host[:-4]
 
         self.base_url = host
-        self.username = headphones.UTORRENT_USERNAME
+        self.username = headphones.CONFIG.UTORRENT_USERNAME
-        self.password = headphones.UTORRENT_PASSWORD
+        self.password = headphones.CONFIG.UTORRENT_PASSWORD
         self.opener = self._make_opener('uTorrent', self.base_url, self.username, self.password)
         self.token = self._get_token()
         #TODO refresh token, when necessary
@@ -48,7 +55,7 @@ class utorrentclient(object):
     def _make_opener(self, realm, base_url, username, password):
         """uTorrent API need HTTP Basic Auth and cookie support for token verify."""
         auth = urllib2.HTTPBasicAuthHandler()
-        auth.add_password(realm=realm,uri=base_url,user=username,passwd=password)
+        auth.add_password(realm=realm, uri=base_url, user=username, passwd=password)
         opener = urllib2.build_opener(auth)
         urllib2.install_opener(opener)
 
@@ -132,7 +139,7 @@ class utorrentclient(object):
             return settings[key]
         return settings
 
-    def remove(self, hash, remove_data = False):
+    def remove(self, hash, remove_data=False):
         if remove_data:
             params = [('action', 'removedata'), ('hash', hash)]
         else:
@@ -156,13 +163,15 @@ class utorrentclient(object):
             logger.debug('URL: ' + str(url))
             logger.debug('uTorrent webUI raised the following error: ' + str(err))
 
+
 def labelTorrent(hash):
-    label = headphones.UTORRENT_LABEL
+    label = headphones.CONFIG.UTORRENT_LABEL
     uTorrentClient = utorrentclient()
     if label:
-        uTorrentClient.setprops(hash,'label',label)
+        uTorrentClient.setprops(hash, 'label', label)
 
-def removeTorrent(hash, remove_data = False):
+
+def removeTorrent(hash, remove_data=False):
     uTorrentClient = utorrentclient()
     status, torrentList = uTorrentClient.list()
     torrents = torrentList['torrents']
@@ -177,14 +186,16 @@ def removeTorrent(hash, remove_data = False):
                 return False
     return False
 
+
 def setSeedRatio(hash, ratio):
     uTorrentClient = utorrentclient()
     uTorrentClient.setprops(hash, 'seed_override', '1')
     if ratio != 0:
-        uTorrentClient.setprops(hash,'seed_ratio', ratio * 10)
+        uTorrentClient.setprops(hash, 'seed_ratio', ratio * 10)
     else:
         # TODO passing -1 should be unlimited
-        uTorrentClient.setprops(hash,'seed_ratio', -10)
+        uTorrentClient.setprops(hash, 'seed_ratio', -10)
+
 
 def dirTorrent(hash, cacheid=None, return_name=None):
 
@@ -204,7 +215,7 @@ def dirTorrent(hash, cacheid=None, return_name=None):
     cacheid = torrentList['torrentc']
 
     for torrent in torrents:
-        if (torrent[0].lower() == hash):
+        if torrent[0].upper() == hash:
             if not return_name:
                 return torrent[26], cacheid
             else:
@@ -212,6 +223,7 @@ def dirTorrent(hash, cacheid=None, return_name=None):
 
     return None, None
 
+
 def addTorrent(link, hash):
     uTorrentClient = utorrentclient()
 
@@ -230,7 +242,7 @@ def addTorrent(link, hash):
     # If there's no folder yet then it's probably a magnet, try until folder is populated
     if torrent_folder == active_dir or not torrent_folder:
         tries = 1
-        while (torrent_folder == active_dir or torrent_folder == None) and tries <= 10:
+        while (torrent_folder == active_dir or torrent_folder is None) and tries <= 10:
             tries += 1
             time.sleep(6)
             torrent_folder, cacheid = dirTorrent(hash, cacheid)
@@ -241,8 +253,11 @@ def addTorrent(link, hash):
         return torrent_folder
     else:
         labelTorrent(hash)
+        if headphones.SYS_PLATFORM != "win32":
+            torrent_folder = torrent_folder.replace('\\', '/')
         return os.path.basename(os.path.normpath(torrent_folder))
 
+
 def getSettingsDirectories():
     uTorrentClient = utorrentclient()
     settings = uTorrentClient.get_settings()
@@ -253,4 +268,3 @@ def getSettingsDirectories():
     if 'dir_completed_download' in settings:
         completed = settings['dir_completed_download'][2]
     return active, completed
-

headphones/version.py

@@ -1 +1 @@
-HEADPHONES_VERSION = "master"
+HEADPHONES_VERSION = "master"

headphones/versioncheck.py

@@ -22,10 +22,11 @@ import subprocess
 
 from headphones import logger, version, request
 
+
 def runGit(args):
 
-    if headphones.GIT_PATH:
+    if headphones.CONFIG.GIT_PATH:
-        git_locations = ['"'+headphones.GIT_PATH+'"']
+        git_locations = ['"' + headphones.CONFIG.GIT_PATH + '"']
     else:
         git_locations = ['git']
 
@@ -35,7 +36,7 @@ def runGit(args):
     output = err = None
 
     for cur_git in git_locations:
-        cmd = cur_git+' '+args
+        cmd = cur_git + ' ' + args
 
         try:
             logger.debug('Trying to execute: "' + cmd + '" with shell in ' + headphones.PROG_DIR)
@@ -59,6 +60,7 @@ def runGit(args):
 
     return (output, err)
 
+
 def getVersion():
 
     if version.HEADPHONES_VERSION.startswith('win32build'):
@@ -82,16 +84,16 @@ def getVersion():
             logger.error('Output doesn\'t look like a hash, not using it')
             cur_commit_hash = None
 
-        if headphones.DO_NOT_OVERRIDE_GIT_BRANCH and headphones.GIT_BRANCH:
+        if headphones.CONFIG.DO_NOT_OVERRIDE_GIT_BRANCH and headphones.CONFIG.GIT_BRANCH:
-            branch_name = headphones.GIT_BRANCH
+            branch_name = headphones.CONFIG.GIT_BRANCH
 
         else:
             branch_name, err = runGit('rev-parse --abbrev-ref HEAD')
             branch_name = branch_name
 
-            if not branch_name and headphones.GIT_BRANCH:
+            if not branch_name and headphones.CONFIG.GIT_BRANCH:
-                logger.error('Could not retrieve branch name from git. Falling back to %s' % headphones.GIT_BRANCH)
+                logger.error('Could not retrieve branch name from git. Falling back to %s' % headphones.CONFIG.GIT_BRANCH)
-                branch_name = headphones.GIT_BRANCH
+                branch_name = headphones.CONFIG.GIT_BRANCH
             if not branch_name:
                 logger.error('Could not retrieve branch name from git. Defaulting to master')
                 branch_name = 'master'
@@ -111,16 +113,17 @@ def getVersion():
             current_version = f.read().strip(' \n\r')
 
         if current_version:
-            return current_version, headphones.GIT_BRANCH
+            return current_version, headphones.CONFIG.GIT_BRANCH
         else:
             return None, 'master'
 
+
 def checkGithub():
     headphones.COMMITS_BEHIND = 0
 
     # Get the latest version available from github
     logger.info('Retrieving latest version information from GitHub')
-    url = 'https://api.github.com/repos/%s/headphones/commits/%s' % (headphones.GIT_USER, headphones.GIT_BRANCH)
+    url = 'https://api.github.com/repos/%s/headphones/commits/%s' % (headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
     version = request.request_json(url, timeout=20, validator=lambda x: type(x) == dict)
 
     if version is None:
@@ -140,7 +143,7 @@ def checkGithub():
         return headphones.LATEST_VERSION
 
     logger.info('Comparing currently installed version with latest GitHub version')
-    url = 'https://api.github.com/repos/%s/headphones/compare/%s...%s' % (headphones.GIT_USER, headphones.LATEST_VERSION, headphones.CURRENT_VERSION)
+    url = 'https://api.github.com/repos/%s/headphones/compare/%s...%s' % (headphones.CONFIG.GIT_USER, headphones.LATEST_VERSION, headphones.CURRENT_VERSION)
     commits = request.request_json(url, timeout=20, whitelist_status_code=404, validator=lambda x: type(x) == dict)
 
     if commits is None:
@@ -161,12 +164,13 @@ def checkGithub():
 
     return headphones.LATEST_VERSION
 
+
 def update():
     if headphones.INSTALL_TYPE == 'win':
         logger.info('Windows .exe updating not supported yet.')
 
     elif headphones.INSTALL_TYPE == 'git':
-        output, err = runGit('pull origin ' + headphones.GIT_BRANCH)
+        output, err = runGit('pull origin ' + headphones.CONFIG.GIT_BRANCH)
 
         if not output:
             logger.error('Couldn\'t download latest version')
@@ -177,22 +181,22 @@ def update():
                 logger.info('No update available, not updating')
                 logger.info('Output: ' + str(output))
             elif line.endswith('Aborting.'):
-                logger.error('Unable to update from git: '+line)
+                logger.error('Unable to update from git: ' + line)
                 logger.info('Output: ' + str(output))
 
     else:
-        tar_download_url = 'https://github.com/%s/headphones/tarball/%s' % (headphones.GIT_USER, headphones.GIT_BRANCH)
+        tar_download_url = 'https://github.com/%s/headphones/tarball/%s' % (headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
         update_dir = os.path.join(headphones.PROG_DIR, 'update')
         version_path = os.path.join(headphones.PROG_DIR, 'version.txt')
 
-        logger.info('Downloading update from: '+ tar_download_url)
+        logger.info('Downloading update from: ' + tar_download_url)
         data = request.request_content(tar_download_url)
 
         if not data:
             logger.error("Unable to retrieve new version from '%s', can't update", tar_download_url)
             return
 
-        download_name = headphones.GIT_BRANCH + '-github'
+        download_name = headphones.CONFIG.GIT_BRANCH + '-github'
         tar_download_path = os.path.join(headphones.PROG_DIR, download_name)
 
         # Save tar to disk
@@ -212,13 +216,13 @@ def update():
         # Find update dir name
         update_dir_contents = [x for x in os.listdir(update_dir) if os.path.isdir(os.path.join(update_dir, x))]
         if len(update_dir_contents) != 1:
-            logger.error("Invalid update data, update failed: "+str(update_dir_contents))
+            logger.error("Invalid update data, update failed: " + str(update_dir_contents))
             return
         content_dir = os.path.join(update_dir, update_dir_contents[0])
 
         # walk temp folder and move files to main folder
         for dirname, dirnames, filenames in os.walk(content_dir):
-            dirname = dirname[len(content_dir)+1:]
+            dirname = dirname[len(content_dir) + 1:]
             for curfile in filenames:
                 old_path = os.path.join(content_dir, dirname, curfile)
                 new_path = os.path.join(headphones.PROG_DIR, dirname, curfile)
@@ -232,6 +236,8 @@ def update():
             with open(version_path, 'w') as f:
                 f.write(str(headphones.LATEST_VERSION))
         except IOError as e:
-            logger.error("Unable to write current version to version.txt, " \
+            logger.error(
-                "update not complete: ", e)
+                "Unable to write current version to version.txt, update not complete: %s",
+                e
+            )
             return

headphones/webstart.py

@@ -22,9 +22,8 @@ from headphones import logger
 from headphones.webserve import WebInterface
 from headphones.helpers import create_https_certificates
 
-def initialize(options=None):
+
-    if options is None:
+def initialize(options):
-        options = {}
 
     # HTTPS stuff stolen from sickbeard
     enable_https = options['enable_https']
@@ -32,16 +31,17 @@ def initialize(options=None):
     https_key = options['https_key']
 
     if enable_https:
-        # If either the HTTPS certificate or key do not exist, make some self-signed ones.
+        # If either the HTTPS certificate or key do not exist, try to make
+        # self-signed ones.
         if not (https_cert and os.path.exists(https_cert)) or not (https_key and os.path.exists(https_key)):
             if not create_https_certificates(https_cert, https_key):
-                logger.warn(u"Unable to create cert/key files, disabling HTTPS")
+                logger.warn("Unable to create certificate and key. Disabling " \
-                headphones.ENABLE_HTTPS = False
+                    "HTTPS")
                 enable_https = False
 
         if not (os.path.exists(https_cert) and os.path.exists(https_key)):
-            logger.warn(u"Disabled HTTPS because of missing CERT and KEY files")
+            logger.warn("Disabled HTTPS because of missing certificate and " \
-            headphones.ENABLE_HTTPS = False
+                "key.")
             enable_https = False
 
     options_dict = {
@@ -52,7 +52,7 @@ def initialize(options=None):
         'tools.encode.encoding': 'utf-8',
         'tools.decode.on': True,
         'log.screen': False,
-        'engine.autoreload_on': False,
+        'engine.autoreload.on': False,
     }
 
     if enable_https:
@@ -71,30 +71,30 @@ def initialize(options=None):
             'tools.staticdir.root': os.path.join(headphones.PROG_DIR, 'data'),
             'tools.proxy.on': options['http_proxy']  # pay attention to X-Forwarded-Proto header
         },
-        '/interfaces':{
+        '/interfaces': {
             'tools.staticdir.on': True,
             'tools.staticdir.dir': "interfaces"
         },
-        '/images':{
+        '/images': {
             'tools.staticdir.on': True,
             'tools.staticdir.dir': "images"
         },
-        '/css':{
+        '/css': {
             'tools.staticdir.on': True,
             'tools.staticdir.dir': "css"
         },
-        '/js':{
+        '/js': {
             'tools.staticdir.on': True,
             'tools.staticdir.dir': "js"
         },
-        '/favicon.ico':{
+        '/favicon.ico': {
             'tools.staticfile.on': True,
             'tools.staticfile.filename': os.path.join(os.path.abspath(
                 os.curdir), "images" + os.sep + "favicon.ico")
         },
-        '/cache':{
+        '/cache': {
             'tools.staticdir.on': True,
-            'tools.staticdir.dir': headphones.CACHE_DIR
+            'tools.staticdir.dir': headphones.CONFIG.CACHE_DIR
         }
     }
 
@@ -104,23 +104,21 @@ def initialize(options=None):
         conf['/'].update({
             'tools.auth_basic.on': True,
             'tools.auth_basic.realm': 'Headphones web server',
-            'tools.auth_basic.checkpassword':  cherrypy.lib.auth_basic \
+            'tools.auth_basic.checkpassword': cherrypy.lib.auth_basic.checkpassword_dict({
-                .checkpassword_dict({
+                options['http_username']: options['http_password']
-                    options['http_username']: options['http_password']
+            })
-                })
         })
-        conf['/api'] = { 'tools.auth_basic.on': False }
+        conf['/api'] = {'tools.auth_basic.on': False}
-
 
     # Prevent time-outs
     cherrypy.engine.timeout_monitor.unsubscribe()
-    cherrypy.tree.mount(WebInterface(), options['http_root'], config=conf)
+    cherrypy.tree.mount(WebInterface(), str(options['http_root']), config=conf)
 
     try:
-        cherrypy.process.servers.check_port(options['http_host'], options['http_port'])
+        cherrypy.process.servers.check_port(str(options['http_host']), options['http_port'])
         cherrypy.server.start()
     except IOError:
         sys.stderr.write('Failed to start on port: %i. Is something else running?\n' % (options['http_port']))
         sys.exit(1)
 
-    cherrypy.server.wait()
+    cherrypy.server.wait()

lib/apscheduler/__init__.py

@@ -1,3 +1,5 @@
-version_info = (2, 0, 0, 'rc', 2)
+version_info = (3, 0, 1)
-version = '.'.join(str(n) for n in version_info[:3])
+version = '3.0.1'
-release = version + ''.join(str(n) for n in version_info[3:])
+release = '3.0.1'
+
+__version__ = release  # PEP 396

lib/apscheduler/events.py

@@ -1,63 +1,72 @@
-__all__ = ('EVENT_SCHEDULER_START', 'EVENT_SCHEDULER_SHUTDOWN',
+__all__ = ('EVENT_SCHEDULER_START', 'EVENT_SCHEDULER_SHUTDOWN', 'EVENT_EXECUTOR_ADDED', 'EVENT_EXECUTOR_REMOVED',
-           'EVENT_JOBSTORE_ADDED', 'EVENT_JOBSTORE_REMOVED',
+           'EVENT_JOBSTORE_ADDED', 'EVENT_JOBSTORE_REMOVED', 'EVENT_ALL_JOBS_REMOVED', 'EVENT_JOB_ADDED',
-           'EVENT_JOBSTORE_JOB_ADDED', 'EVENT_JOBSTORE_JOB_REMOVED',
+           'EVENT_JOB_REMOVED', 'EVENT_JOB_MODIFIED', 'EVENT_JOB_EXECUTED', 'EVENT_JOB_ERROR', 'EVENT_JOB_MISSED',
-           'EVENT_JOB_EXECUTED', 'EVENT_JOB_ERROR', 'EVENT_JOB_MISSED',
+           'SchedulerEvent', 'JobEvent', 'JobExecutionEvent')
-           'EVENT_ALL', 'SchedulerEvent', 'JobStoreEvent', 'JobEvent')
 
 
-EVENT_SCHEDULER_START = 1        # The scheduler was started
+EVENT_SCHEDULER_START = 1
-EVENT_SCHEDULER_SHUTDOWN = 2     # The scheduler was shut down
+EVENT_SCHEDULER_SHUTDOWN = 2
-EVENT_JOBSTORE_ADDED = 4         # A job store was added to the scheduler
+EVENT_EXECUTOR_ADDED = 4
-EVENT_JOBSTORE_REMOVED = 8       # A job store was removed from the scheduler
+EVENT_EXECUTOR_REMOVED = 8
-EVENT_JOBSTORE_JOB_ADDED = 16    # A job was added to a job store
+EVENT_JOBSTORE_ADDED = 16
-EVENT_JOBSTORE_JOB_REMOVED = 32  # A job was removed from a job store
+EVENT_JOBSTORE_REMOVED = 32
-EVENT_JOB_EXECUTED = 64          # A job was executed successfully
+EVENT_ALL_JOBS_REMOVED = 64
-EVENT_JOB_ERROR = 128            # A job raised an exception during execution
+EVENT_JOB_ADDED = 128
-EVENT_JOB_MISSED = 256           # A job's execution was missed
+EVENT_JOB_REMOVED = 256
-EVENT_ALL = (EVENT_SCHEDULER_START | EVENT_SCHEDULER_SHUTDOWN |
+EVENT_JOB_MODIFIED = 512
-             EVENT_JOBSTORE_ADDED | EVENT_JOBSTORE_REMOVED |
+EVENT_JOB_EXECUTED = 1024
-             EVENT_JOBSTORE_JOB_ADDED | EVENT_JOBSTORE_JOB_REMOVED |
+EVENT_JOB_ERROR = 2048
-             EVENT_JOB_EXECUTED | EVENT_JOB_ERROR | EVENT_JOB_MISSED)
+EVENT_JOB_MISSED = 4096
+EVENT_ALL = (EVENT_SCHEDULER_START | EVENT_SCHEDULER_SHUTDOWN | EVENT_JOBSTORE_ADDED | EVENT_JOBSTORE_REMOVED |
+             EVENT_JOB_ADDED | EVENT_JOB_REMOVED | EVENT_JOB_MODIFIED | EVENT_JOB_EXECUTED |
+             EVENT_JOB_ERROR | EVENT_JOB_MISSED)
 
 
 class SchedulerEvent(object):
     """
     An event that concerns the scheduler itself.
 
-    :var code: the type code of this event
+    :ivar code: the type code of this event
+    :ivar alias: alias of the job store or executor that was added or removed (if applicable)
     """
-    def __init__(self, code):
+
+    def __init__(self, code, alias=None):
+        super(SchedulerEvent, self).__init__()
         self.code = code
-
-
-class JobStoreEvent(SchedulerEvent):
-    """
-    An event that concerns job stores.
-
-    :var alias: the alias of the job store involved
-    :var job: the new job if a job was added
-    """
-    def __init__(self, code, alias, job=None):
-        SchedulerEvent.__init__(self, code)
         self.alias = alias
-        if job:
+
-            self.job = job
+    def __repr__(self):
+        return '<%s (code=%d)>' % (self.__class__.__name__, self.code)
 
 
 class JobEvent(SchedulerEvent):
+    """
+    An event that concerns a job.
+
+    :ivar code: the type code of this event
+    :ivar job_id: identifier of the job in question
+    :ivar jobstore: alias of the job store containing the job in question
+    """
+
+    def __init__(self, code, job_id, jobstore):
+        super(JobEvent, self).__init__(code)
+        self.code = code
+        self.job_id = job_id
+        self.jobstore = jobstore
+
+
+class JobExecutionEvent(JobEvent):
     """
     An event that concerns the execution of individual jobs.
 
-    :var job: the job instance in question
+    :ivar scheduled_run_time: the time when the job was scheduled to be run
-    :var scheduled_run_time: the time when the job was scheduled to be run
+    :ivar retval: the return value of the successfully executed job
-    :var retval: the return value of the successfully executed job
+    :ivar exception: the exception raised by the job
-    :var exception: the exception raised by the job
+    :ivar traceback: a formatted traceback for the exception
-    :var traceback: the traceback object associated with the exception
     """
-    def __init__(self, code, job, scheduled_run_time, retval=None,
+
-                 exception=None, traceback=None):
+    def __init__(self, code, job_id, jobstore, scheduled_run_time, retval=None, exception=None, traceback=None):
-        SchedulerEvent.__init__(self, code)
+        super(JobExecutionEvent, self).__init__(code, job_id, jobstore)
-        self.job = job
         self.scheduled_run_time = scheduled_run_time
         self.retval = retval
         self.exception = exception

lib/apscheduler/executors/asyncio.py

@@ -0,0 +1,28 @@
+from __future__ import absolute_import
+import sys
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class AsyncIOExecutor(BaseExecutor):
+    """
+    Runs jobs in the default executor of the event loop.
+
+    Plugin alias: ``asyncio``
+    """
+
+    def start(self, scheduler, alias):
+        super(AsyncIOExecutor, self).start(scheduler, alias)
+        self._eventloop = scheduler._eventloop
+
+    def _do_submit_job(self, job, run_times):
+        def callback(f):
+            try:
+                events = f.result()
+            except:
+                self._run_job_error(job.id, *sys.exc_info()[1:])
+            else:
+                self._run_job_success(job.id, events)
+
+        f = self._eventloop.run_in_executor(None, run_job, job, job._jobstore_alias, run_times, self._logger.name)
+        f.add_done_callback(callback)

lib/apscheduler/executors/base.py

@@ -0,0 +1,119 @@
+from abc import ABCMeta, abstractmethod
+from collections import defaultdict
+from datetime import datetime, timedelta
+from traceback import format_tb
+import logging
+import sys
+
+from pytz import utc
+import six
+
+from apscheduler.events import JobExecutionEvent, EVENT_JOB_MISSED, EVENT_JOB_ERROR, EVENT_JOB_EXECUTED
+
+
+class MaxInstancesReachedError(Exception):
+    def __init__(self, job):
+        super(MaxInstancesReachedError, self).__init__(
+            'Job "%s" has already reached its maximum number of instances (%d)' % (job.id, job.max_instances))
+
+
+class BaseExecutor(six.with_metaclass(ABCMeta, object)):
+    """Abstract base class that defines the interface that every executor must implement."""
+
+    _scheduler = None
+    _lock = None
+    _logger = logging.getLogger('apscheduler.executors')
+
+    def __init__(self):
+        super(BaseExecutor, self).__init__()
+        self._instances = defaultdict(lambda: 0)
+
+    def start(self, scheduler, alias):
+        """
+        Called by the scheduler when the scheduler is being started or when the executor is being added to an already
+        running scheduler.
+
+        :param apscheduler.schedulers.base.BaseScheduler scheduler: the scheduler that is starting this executor
+        :param str|unicode alias: alias of this executor as it was assigned to the scheduler
+        """
+
+        self._scheduler = scheduler
+        self._lock = scheduler._create_lock()
+        self._logger = logging.getLogger('apscheduler.executors.%s' % alias)
+
+    def shutdown(self, wait=True):
+        """
+        Shuts down this executor.
+
+        :param bool wait: ``True`` to wait until all submitted jobs have been executed
+        """
+
+    def submit_job(self, job, run_times):
+        """
+        Submits job for execution.
+
+        :param Job job: job to execute
+        :param list[datetime] run_times: list of datetimes specifying when the job should have been run
+        :raises MaxInstancesReachedError: if the maximum number of allowed instances for this job has been reached
+        """
+
+        assert self._lock is not None, 'This executor has not been started yet'
+        with self._lock:
+            if self._instances[job.id] >= job.max_instances:
+                raise MaxInstancesReachedError(job)
+
+            self._do_submit_job(job, run_times)
+            self._instances[job.id] += 1
+
+    @abstractmethod
+    def _do_submit_job(self, job, run_times):
+        """Performs the actual task of scheduling `run_job` to be called."""
+
+    def _run_job_success(self, job_id, events):
+        """Called by the executor with the list of generated events when `run_job` has been successfully called."""
+
+        with self._lock:
+            self._instances[job_id] -= 1
+
+        for event in events:
+            self._scheduler._dispatch_event(event)
+
+    def _run_job_error(self, job_id, exc, traceback=None):
+        """Called by the executor with the exception if there is an error calling `run_job`."""
+
+        with self._lock:
+            self._instances[job_id] -= 1
+
+        exc_info = (exc.__class__, exc, traceback)
+        self._logger.error('Error running job %s', job_id, exc_info=exc_info)
+
+
+def run_job(job, jobstore_alias, run_times, logger_name):
+    """Called by executors to run the job. Returns a list of scheduler events to be dispatched by the scheduler."""
+
+    events = []
+    logger = logging.getLogger(logger_name)
+    for run_time in run_times:
+        # See if the job missed its run time window, and handle possible misfires accordingly
+        if job.misfire_grace_time is not None:
+            difference = datetime.now(utc) - run_time
+            grace_time = timedelta(seconds=job.misfire_grace_time)
+            if difference > grace_time:
+                events.append(JobExecutionEvent(EVENT_JOB_MISSED, job.id, jobstore_alias, run_time))
+                logger.warning('Run time of job "%s" was missed by %s', job, difference)
+                continue
+
+        logger.info('Running job "%s" (scheduled at %s)', job, run_time)
+        try:
+            retval = job.func(*job.args, **job.kwargs)
+        except:
+            exc, tb = sys.exc_info()[1:]
+            formatted_tb = ''.join(format_tb(tb))
+            events.append(JobExecutionEvent(EVENT_JOB_ERROR, job.id, jobstore_alias, run_time, exception=exc,
+                                            traceback=formatted_tb))
+            logger.exception('Job "%s" raised an exception', job)
+        else:
+            events.append(JobExecutionEvent(EVENT_JOB_EXECUTED, job.id, jobstore_alias, run_time, retval=retval))
+            logger.info('Job "%s" executed successfully', job)
+
+    return events

lib/apscheduler/executors/debug.py

@@ -0,0 +1,19 @@
+import sys
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class DebugExecutor(BaseExecutor):
+    """
+    A special executor that executes the target callable directly instead of deferring it to a thread or process.
+
+    Plugin alias: ``debug``
+    """
+
+    def _do_submit_job(self, job, run_times):
+        try:
+            events = run_job(job, job._jobstore_alias, run_times, self._logger.name)
+        except:
+            self._run_job_error(job.id, *sys.exc_info()[1:])
+        else:
+            self._run_job_success(job.id, events)

lib/apscheduler/executors/gevent.py

@@ -0,0 +1,29 @@
+from __future__ import absolute_import
+import sys
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+try:
+    import gevent
+except ImportError:  # pragma: nocover
+    raise ImportError('GeventExecutor requires gevent installed')
+
+
+class GeventExecutor(BaseExecutor):
+    """
+    Runs jobs as greenlets.
+
+    Plugin alias: ``gevent``
+    """
+
+    def _do_submit_job(self, job, run_times):
+        def callback(greenlet):
+            try:
+                events = greenlet.get()
+            except:
+                self._run_job_error(job.id, *sys.exc_info()[1:])
+            else:
+                self._run_job_success(job.id, events)
+
+        gevent.spawn(run_job, job, job._jobstore_alias, run_times, self._logger.name).link(callback)

lib/apscheduler/executors/pool.py

@@ -0,0 +1,54 @@
+from abc import abstractmethod
+import concurrent.futures
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class BasePoolExecutor(BaseExecutor):
+    @abstractmethod
+    def __init__(self, pool):
+        super(BasePoolExecutor, self).__init__()
+        self._pool = pool
+
+    def _do_submit_job(self, job, run_times):
+        def callback(f):
+            exc, tb = (f.exception_info() if hasattr(f, 'exception_info') else
+                       (f.exception(), getattr(f.exception(), '__traceback__', None)))
+            if exc:
+                self._run_job_error(job.id, exc, tb)
+            else:
+                self._run_job_success(job.id, f.result())
+
+        f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
+        f.add_done_callback(callback)
+
+    def shutdown(self, wait=True):
+        self._pool.shutdown(wait)
+
+
+class ThreadPoolExecutor(BasePoolExecutor):
+    """
+    An executor that runs jobs in a concurrent.futures thread pool.
+
+    Plugin alias: ``threadpool``
+
+    :param max_workers: the maximum number of spawned threads.
+    """
+
+    def __init__(self, max_workers=10):
+        pool = concurrent.futures.ThreadPoolExecutor(int(max_workers))
+        super(ThreadPoolExecutor, self).__init__(pool)
+
+
+class ProcessPoolExecutor(BasePoolExecutor):
+    """
+    An executor that runs jobs in a concurrent.futures process pool.
+
+    Plugin alias: ``processpool``
+
+    :param max_workers: the maximum number of spawned processes.
+    """
+
+    def __init__(self, max_workers=10):
+        pool = concurrent.futures.ProcessPoolExecutor(int(max_workers))
+        super(ProcessPoolExecutor, self).__init__(pool)

lib/apscheduler/executors/twisted.py

@@ -0,0 +1,25 @@
+from __future__ import absolute_import
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class TwistedExecutor(BaseExecutor):
+    """
+    Runs jobs in the reactor's thread pool.
+
+    Plugin alias: ``twisted``
+    """
+
+    def start(self, scheduler, alias):
+        super(TwistedExecutor, self).start(scheduler, alias)
+        self._reactor = scheduler._reactor
+
+    def _do_submit_job(self, job, run_times):
+        def callback(success, result):
+            if success:
+                self._run_job_success(job.id, result)
+            else:
+                self._run_job_error(job.id, result.value, result.tb)
+
+        self._reactor.getThreadPool().callInThreadWithCallback(callback, run_job, job, job._jobstore_alias, run_times,
+                                                               self._logger.name)

lib/apscheduler/job.py

@@ -1,134 +1,252 @@
-"""
+from collections import Iterable, Mapping
-Jobs represent scheduled tasks.
+from uuid import uuid4
-"""
 
-from threading import Lock
+import six
-from datetime import timedelta
 
-from apscheduler.util import to_unicode, ref_to_obj, get_callable_name,\
+from apscheduler.triggers.base import BaseTrigger
-    obj_to_ref
+from apscheduler.util import ref_to_obj, obj_to_ref, datetime_repr, repr_escape, get_callable_name, check_callable_args, \
-
+    convert_to_datetime
-
-class MaxInstancesReachedError(Exception):
-    pass
 
 
 class Job(object):
     """
-    Encapsulates the actual Job along with its metadata. Job instances
+    Contains the options given when scheduling callables and its current schedule and other state.
-    are created by the scheduler when adding jobs, and it should not be
+    This class should never be instantiated by the user.
-    directly instantiated.
 
-    :param trigger: trigger that determines the execution times
+    :var str id: the unique identifier of this job
-    :param func: callable to call when the trigger is triggered
+    :var str name: the description of this job
-    :param args: list of positional arguments to call func with
+    :var func: the callable to execute
-    :param kwargs: dict of keyword arguments to call func with
+    :var tuple|list args: positional arguments to the callable
-    :param name: name of the job (optional)
+    :var dict kwargs: keyword arguments to the callable
-    :param misfire_grace_time: seconds after the designated run time that
+    :var bool coalesce: whether to only run the job once when several run times are due
-        the job is still allowed to be run
+    :var trigger: the trigger object that controls the schedule of this job
-    :param coalesce: run once instead of many times if the scheduler determines
+    :var str executor: the name of the executor that will run this job
-        that the job should be run more than once in succession
+    :var int misfire_grace_time: the time (in seconds) how much this job's execution is allowed to be late
-    :param max_runs: maximum number of times this job is allowed to be
+    :var int max_instances: the maximum number of concurrently executing instances allowed for this job
-        triggered
+    :var datetime.datetime next_run_time: the next scheduled run time of this job
-    :param max_instances: maximum number of concurrently running
-        instances allowed for this job
     """
-    id = None
-    next_run_time = None
 
-    def __init__(self, trigger, func, args, kwargs, misfire_grace_time,
+    __slots__ = ('_scheduler', '_jobstore_alias', 'id', 'trigger', 'executor', 'func', 'func_ref', 'args', 'kwargs',
-                 coalesce, name=None, max_runs=None, max_instances=1):
+                 'name', 'misfire_grace_time', 'coalesce', 'max_instances', 'next_run_time')
-        if not trigger:
-            raise ValueError('The trigger must not be None')
-        if not hasattr(func, '__call__'):
-            raise TypeError('func must be callable')
-        if not hasattr(args, '__getitem__'):
-            raise TypeError('args must be a list-like object')
-        if not hasattr(kwargs, '__getitem__'):
-            raise TypeError('kwargs must be a dict-like object')
-        if misfire_grace_time <= 0:
-            raise ValueError('misfire_grace_time must be a positive value')
-        if max_runs is not None and max_runs <= 0:
-            raise ValueError('max_runs must be a positive value')
-        if max_instances <= 0:
-            raise ValueError('max_instances must be a positive value')
 
-        self._lock = Lock()
+    def __init__(self, scheduler, id=None, **kwargs):
+        super(Job, self).__init__()
+        self._scheduler = scheduler
+        self._jobstore_alias = None
+        self._modify(id=id or uuid4().hex, **kwargs)
 
-        self.trigger = trigger
+    def modify(self, **changes):
-        self.func = func
-        self.args = args
-        self.kwargs = kwargs
-        self.name = to_unicode(name or get_callable_name(func))
-        self.misfire_grace_time = misfire_grace_time
-        self.coalesce = coalesce
-        self.max_runs = max_runs
-        self.max_instances = max_instances
-        self.runs = 0
-        self.instances = 0
-
-    def compute_next_run_time(self, now):
-        if self.runs == self.max_runs:
-            self.next_run_time = None
-        else:
-            self.next_run_time = self.trigger.get_next_fire_time(now)
-
-        return self.next_run_time
-
-    def get_run_times(self, now):
         """
-        Computes the scheduled run times between ``next_run_time`` and ``now``.
+        Makes the given changes to this job and saves it in the associated job store.
+        Accepted keyword arguments are the same as the variables on this class.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.modify_job`
         """
+
+        self._scheduler.modify_job(self.id, self._jobstore_alias, **changes)
+
+    def reschedule(self, trigger, **trigger_args):
+        """
+        Shortcut for switching the trigger on this job.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.reschedule_job`
+        """
+
+        self._scheduler.reschedule_job(self.id, self._jobstore_alias, trigger, **trigger_args)
+
+    def pause(self):
+        """
+        Temporarily suspend the execution of this job.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.pause_job`
+        """
+
+        self._scheduler.pause_job(self.id, self._jobstore_alias)
+
+    def resume(self):
+        """
+        Resume the schedule of this job if previously paused.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.resume_job`
+        """
+
+        self._scheduler.resume_job(self.id, self._jobstore_alias)
+
+    def remove(self):
+        """
+        Unschedules this job and removes it from its associated job store.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.remove_job`
+        """
+
+        self._scheduler.remove_job(self.id, self._jobstore_alias)
+
+    @property
+    def pending(self):
+        """Returns ``True`` if the referenced job is still waiting to be added to its designated job store."""
+
+        return self._jobstore_alias is None
+
+    #
+    # Private API
+    #
+
+    def _get_run_times(self, now):
+        """
+        Computes the scheduled run times between ``next_run_time`` and ``now`` (inclusive).
+
+        :type now: datetime.datetime
+        :rtype: list[datetime.datetime]
+        """
+
         run_times = []
-        run_time = self.next_run_time
+        next_run_time = self.next_run_time
-        increment = timedelta(microseconds=1)
+        while next_run_time and next_run_time <= now:
-        while ((not self.max_runs or self.runs < self.max_runs) and
+            run_times.append(next_run_time)
-               run_time and run_time <= now):
+            next_run_time = self.trigger.get_next_fire_time(next_run_time, now)
-            run_times.append(run_time)
-            run_time = self.trigger.get_next_fire_time(run_time + increment)
 
         return run_times
 
-    def add_instance(self):
+    def _modify(self, **changes):
-        self._lock.acquire()
+        """Validates the changes to the Job and makes the modifications if and only if all of them validate."""
-        try:
-            if self.instances == self.max_instances:
-                raise MaxInstancesReachedError
-            self.instances += 1
-        finally:
-            self._lock.release()
 
-    def remove_instance(self):
+        approved = {}
-        self._lock.acquire()
+
-        try:
+        if 'id' in changes:
-            assert self.instances > 0, 'Already at 0 instances'
+            value = changes.pop('id')
-            self.instances -= 1
+            if not isinstance(value, six.string_types):
-        finally:
+                raise TypeError("id must be a nonempty string")
-            self._lock.release()
+            if hasattr(self, 'id'):
+                raise ValueError('The job ID may not be changed')
+            approved['id'] = value
+
+        if 'func' in changes or 'args' in changes or 'kwargs' in changes:
+            func = changes.pop('func') if 'func' in changes else self.func
+            args = changes.pop('args') if 'args' in changes else self.args
+            kwargs = changes.pop('kwargs') if 'kwargs' in changes else self.kwargs
+
+            if isinstance(func, str):
+                func_ref = func
+                func = ref_to_obj(func)
+            elif callable(func):
+                try:
+                    func_ref = obj_to_ref(func)
+                except ValueError:
+                    # If this happens, this Job won't be serializable
+                    func_ref = None
+            else:
+                raise TypeError('func must be a callable or a textual reference to one')
+
+            if not hasattr(self, 'name') and changes.get('name', None) is None:
+                changes['name'] = get_callable_name(func)
+
+            if isinstance(args, six.string_types) or not isinstance(args, Iterable):
+                raise TypeError('args must be a non-string iterable')
+            if isinstance(kwargs, six.string_types) or not isinstance(kwargs, Mapping):
+                raise TypeError('kwargs must be a dict-like object')
+
+            check_callable_args(func, args, kwargs)
+
+            approved['func'] = func
+            approved['func_ref'] = func_ref
+            approved['args'] = args
+            approved['kwargs'] = kwargs
+
+        if 'name' in changes:
+            value = changes.pop('name')
+            if not value or not isinstance(value, six.string_types):
+                raise TypeError("name must be a nonempty string")
+            approved['name'] = value
+
+        if 'misfire_grace_time' in changes:
+            value = changes.pop('misfire_grace_time')
+            if value is not None and (not isinstance(value, six.integer_types) or value <= 0):
+                raise TypeError('misfire_grace_time must be either None or a positive integer')
+            approved['misfire_grace_time'] = value
+
+        if 'coalesce' in changes:
+            value = bool(changes.pop('coalesce'))
+            approved['coalesce'] = value
+
+        if 'max_instances' in changes:
+            value = changes.pop('max_instances')
+            if not isinstance(value, six.integer_types) or value <= 0:
+                raise TypeError('max_instances must be a positive integer')
+            approved['max_instances'] = value
+
+        if 'trigger' in changes:
+            trigger = changes.pop('trigger')
+            if not isinstance(trigger, BaseTrigger):
+                raise TypeError('Expected a trigger instance, got %s instead' % trigger.__class__.__name__)
+
+            approved['trigger'] = trigger
+
+        if 'executor' in changes:
+            value = changes.pop('executor')
+            if not isinstance(value, six.string_types):
+                raise TypeError('executor must be a string')
+            approved['executor'] = value
+
+        if 'next_run_time' in changes:
+            value = changes.pop('next_run_time')
+            approved['next_run_time'] = convert_to_datetime(value, self._scheduler.timezone, 'next_run_time')
+
+        if changes:
+            raise AttributeError('The following are not modifiable attributes of Job: %s' % ', '.join(changes))
+
+        for key, value in six.iteritems(approved):
+            setattr(self, key, value)
 
     def __getstate__(self):
-        # Prevents the unwanted pickling of transient or unpicklable variables
+        # Don't allow this Job to be serialized if the function reference could not be determined
-        state = self.__dict__.copy()
+        if not self.func_ref:
-        state.pop('instances', None)
+            raise ValueError('This Job cannot be serialized since the reference to its callable (%r) could not be '
-        state.pop('func', None)
+                             'determined. Consider giving a textual reference (module:function name) instead.' %
-        state.pop('_lock', None)
+                             (self.func,))
-        state['func_ref'] = obj_to_ref(self.func)
+
-        return state
+        return {
+            'version': 1,
+            'id': self.id,
+            'func': self.func_ref,
+            'trigger': self.trigger,
+            'executor': self.executor,
+            'args': self.args,
+            'kwargs': self.kwargs,
+            'name': self.name,
+            'misfire_grace_time': self.misfire_grace_time,
+            'coalesce': self.coalesce,
+            'max_instances': self.max_instances,
+            'next_run_time': self.next_run_time
+        }
 
     def __setstate__(self, state):
-        state['instances'] = 0
+        if state.get('version', 1) > 1:
-        state['func'] = ref_to_obj(state.pop('func_ref'))
+            raise ValueError('Job has version %s, but only version 1 can be handled' % state['version'])
-        state['_lock'] = Lock()
+
-        self.__dict__ = state
+        self.id = state['id']
+        self.func_ref = state['func']
+        self.func = ref_to_obj(self.func_ref)
+        self.trigger = state['trigger']
+        self.executor = state['executor']
+        self.args = state['args']
+        self.kwargs = state['kwargs']
+        self.name = state['name']
+        self.misfire_grace_time = state['misfire_grace_time']
+        self.coalesce = state['coalesce']
+        self.max_instances = state['max_instances']
+        self.next_run_time = state['next_run_time']
 
     def __eq__(self, other):
         if isinstance(other, Job):
-            return self.id is not None and other.id == self.id or self is other
+            return self.id == other.id
         return NotImplemented
 
     def __repr__(self):
-        return '<Job (name=%s, trigger=%s)>' % (self.name, repr(self.trigger))
+        return '<Job (id=%s name=%s)>' % (repr_escape(self.id), repr_escape(self.name))
 
     def __str__(self):
-        return '%s (trigger: %s, next run at: %s)' % (self.name,
+        return '%s (trigger: %s, next run at: %s)' % (repr_escape(self.name), repr_escape(str(self.trigger)),
-            str(self.trigger), str(self.next_run_time))
+                                                      datetime_repr(self.next_run_time))
+
+    def __unicode__(self):
+        return six.u('%s (trigger: %s, next run at: %s)') % (self.name, self.trigger, datetime_repr(self.next_run_time))

lib/apscheduler/jobstores/base.py

@@ -1,25 +1,127 @@
-"""
+from abc import ABCMeta, abstractmethod
-Abstract base class that provides the interface needed by all job stores.
+import logging
-Job store methods are also documented here.
+
-"""
+import six
 
 
-class JobStore(object):
+class JobLookupError(KeyError):
-    def add_job(self, job):
+    """Raised when the job store cannot find a job for update or removal."""
-        """Adds the given job from this store."""
-        raise NotImplementedError
 
-    def update_job(self, job):
+    def __init__(self, job_id):
-        """Persists the running state of the given job."""
+        super(JobLookupError, self).__init__(six.u('No job by the id of %s was found') % job_id)
-        raise NotImplementedError
 
-    def remove_job(self, job):
-        """Removes the given jobs from this store."""
-        raise NotImplementedError
 
-    def load_jobs(self):
+class ConflictingIdError(KeyError):
-        """Loads jobs from this store into memory."""
+    """Raised when the uniqueness of job IDs is being violated."""
-        raise NotImplementedError
 
-    def close(self):
+    def __init__(self, job_id):
+        super(ConflictingIdError, self).__init__(six.u('Job identifier (%s) conflicts with an existing job') % job_id)
+
+
+class TransientJobError(ValueError):
+    """Raised when an attempt to add transient (with no func_ref) job to a persistent job store is detected."""
+
+    def __init__(self, job_id):
+        super(TransientJobError, self).__init__(
+            six.u('Job (%s) cannot be added to this job store because a reference to the callable could not be '
+                  'determined.') % job_id)
+
+
+class BaseJobStore(six.with_metaclass(ABCMeta)):
+    """Abstract base class that defines the interface that every job store must implement."""
+
+    _scheduler = None
+    _alias = None
+    _logger = logging.getLogger('apscheduler.jobstores')
+
+    def start(self, scheduler, alias):
+        """
+        Called by the scheduler when the scheduler is being started or when the job store is being added to an already
+        running scheduler.
+
+        :param apscheduler.schedulers.base.BaseScheduler scheduler: the scheduler that is starting this job store
+        :param str|unicode alias: alias of this job store as it was assigned to the scheduler
+        """
+
+        self._scheduler = scheduler
+        self._alias = alias
+        self._logger = logging.getLogger('apscheduler.jobstores.%s' % alias)
+
+    def shutdown(self):
         """Frees any resources still bound to this job store."""
+
+    @abstractmethod
+    def lookup_job(self, job_id):
+        """
+        Returns a specific job, or ``None`` if it isn't found..
+
+        The job store is responsible for setting the ``scheduler`` and ``jobstore`` attributes of the returned job to
+        point to the scheduler and itself, respectively.
+
+        :param str|unicode job_id: identifier of the job
+        :rtype: Job
+        """
+
+    @abstractmethod
+    def get_due_jobs(self, now):
+        """
+        Returns the list of jobs that have ``next_run_time`` earlier or equal to ``now``.
+        The returned jobs must be sorted by next run time (ascending).
+
+        :param datetime.datetime now: the current (timezone aware) datetime
+        :rtype: list[Job]
+        """
+
+    @abstractmethod
+    def get_next_run_time(self):
+        """
+        Returns the earliest run time of all the jobs stored in this job store, or ``None`` if there are no active jobs.
+
+        :rtype: datetime.datetime
+        """
+
+    @abstractmethod
+    def get_all_jobs(self):
+        """
+        Returns a list of all jobs in this job store. The returned jobs should be sorted by next run time (ascending).
+        Paused jobs (next_run_time is None) should be sorted last.
+
+        The job store is responsible for setting the ``scheduler`` and ``jobstore`` attributes of the returned jobs to
+        point to the scheduler and itself, respectively.
+
+        :rtype: list[Job]
+        """
+
+    @abstractmethod
+    def add_job(self, job):
+        """
+        Adds the given job to this store.
+
+        :param Job job: the job to add
+        :raises ConflictingIdError: if there is another job in this store with the same ID
+        """
+
+    @abstractmethod
+    def update_job(self, job):
+        """
+        Replaces the job in the store with the given newer version.
+
+        :param Job job: the job to update
+        :raises JobLookupError: if the job does not exist
+        """
+
+    @abstractmethod
+    def remove_job(self, job_id):
+        """
+        Removes the given job from this store.
+
+        :param str|unicode job_id: identifier of the job
+        :raises JobLookupError: if the job does not exist
+        """
+
+    @abstractmethod
+    def remove_all_jobs(self):
+        """Removes all jobs from this store."""
+
+    def __repr__(self):
+        return '<%s>' % self.__class__.__name__

lib/apscheduler/jobstores/memory.py

@@ -0,0 +1,107 @@
+from __future__ import absolute_import
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import datetime_to_utc_timestamp
+
+
+class MemoryJobStore(BaseJobStore):
+    """
+    Stores jobs in an array in RAM. Provides no persistence support.
+
+    Plugin alias: ``memory``
+    """
+
+    def __init__(self):
+        super(MemoryJobStore, self).__init__()
+        self._jobs = []  # list of (job, timestamp), sorted by next_run_time and job id (ascending)
+        self._jobs_index = {}  # id -> (job, timestamp) lookup table
+
+    def lookup_job(self, job_id):
+        return self._jobs_index.get(job_id, (None, None))[0]
+
+    def get_due_jobs(self, now):
+        now_timestamp = datetime_to_utc_timestamp(now)
+        pending = []
+        for job, timestamp in self._jobs:
+            if timestamp is None or timestamp > now_timestamp:
+                break
+            pending.append(job)
+
+        return pending
+
+    def get_next_run_time(self):
+        return self._jobs[0][0].next_run_time if self._jobs else None
+
+    def get_all_jobs(self):
+        return [j[0] for j in self._jobs]
+
+    def add_job(self, job):
+        if job.id in self._jobs_index:
+            raise ConflictingIdError(job.id)
+
+        timestamp = datetime_to_utc_timestamp(job.next_run_time)
+        index = self._get_job_index(timestamp, job.id)
+        self._jobs.insert(index, (job, timestamp))
+        self._jobs_index[job.id] = (job, timestamp)
+
+    def update_job(self, job):
+        old_job, old_timestamp = self._jobs_index.get(job.id, (None, None))
+        if old_job is None:
+            raise JobLookupError(job.id)
+
+        # If the next run time has not changed, simply replace the job in its present index.
+        # Otherwise, reinsert the job to the list to preserve the ordering.
+        old_index = self._get_job_index(old_timestamp, old_job.id)
+        new_timestamp = datetime_to_utc_timestamp(job.next_run_time)
+        if old_timestamp == new_timestamp:
+            self._jobs[old_index] = (job, new_timestamp)
+        else:
+            del self._jobs[old_index]
+            new_index = self._get_job_index(new_timestamp, job.id)
+            self._jobs.insert(new_index, (job, new_timestamp))
+
+        self._jobs_index[old_job.id] = (job, new_timestamp)
+
+    def remove_job(self, job_id):
+        job, timestamp = self._jobs_index.get(job_id, (None, None))
+        if job is None:
+            raise JobLookupError(job_id)
+
+        index = self._get_job_index(timestamp, job_id)
+        del self._jobs[index]
+        del self._jobs_index[job.id]
+
+    def remove_all_jobs(self):
+        self._jobs = []
+        self._jobs_index = {}
+
+    def shutdown(self):
+        self.remove_all_jobs()
+
+    def _get_job_index(self, timestamp, job_id):
+        """
+        Returns the index of the given job, or if it's not found, the index where the job should be inserted based on
+        the given timestamp.
+
+        :type timestamp: int
+        :type job_id: str
+        """
+
+        lo, hi = 0, len(self._jobs)
+        timestamp = float('inf') if timestamp is None else timestamp
+        while lo < hi:
+            mid = (lo + hi) // 2
+            mid_job, mid_timestamp = self._jobs[mid]
+            mid_timestamp = float('inf') if mid_timestamp is None else mid_timestamp
+            if mid_timestamp > timestamp:
+                hi = mid
+            elif mid_timestamp < timestamp:
+                lo = mid + 1
+            elif mid_job.id > job_id:
+                hi = mid
+            elif mid_job.id < job_id:
+                lo = mid + 1
+            else:
+                return mid
+
+        return lo

lib/apscheduler/jobstores/mongodb.py

@@ -0,0 +1,124 @@
+from __future__ import absolute_import
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import maybe_ref, datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from bson.binary import Binary
+    from pymongo.errors import DuplicateKeyError
+    from pymongo import MongoClient, ASCENDING
+except ImportError:  # pragma: nocover
+    raise ImportError('MongoDBJobStore requires PyMongo installed')
+
+
+class MongoDBJobStore(BaseJobStore):
+    """
+    Stores jobs in a MongoDB database. Any leftover keyword arguments are directly passed to pymongo's `MongoClient
+    <http://api.mongodb.org/python/current/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient>`_.
+
+    Plugin alias: ``mongodb``
+
+    :param str database: database to store jobs in
+    :param str collection: collection to store jobs in
+    :param client: a :class:`~pymongo.mongo_client.MongoClient` instance to use instead of providing connection
+                   arguments
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    """
+
+    def __init__(self, database='apscheduler', collection='jobs', client=None,
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL, **connect_args):
+        super(MongoDBJobStore, self).__init__()
+        self.pickle_protocol = pickle_protocol
+
+        if not database:
+            raise ValueError('The "database" parameter must not be empty')
+        if not collection:
+            raise ValueError('The "collection" parameter must not be empty')
+
+        if client:
+            self.connection = maybe_ref(client)
+        else:
+            connect_args.setdefault('w', 1)
+            self.connection = MongoClient(**connect_args)
+
+        self.collection = self.connection[database][collection]
+        self.collection.ensure_index('next_run_time', sparse=True)
+
+    def lookup_job(self, job_id):
+        document = self.collection.find_one(job_id, ['job_state'])
+        return self._reconstitute_job(document['job_state']) if document else None
+
+    def get_due_jobs(self, now):
+        timestamp = datetime_to_utc_timestamp(now)
+        return self._get_jobs({'next_run_time': {'$lte': timestamp}})
+
+    def get_next_run_time(self):
+        document = self.collection.find_one({'next_run_time': {'$ne': None}}, fields=['next_run_time'],
+                                            sort=[('next_run_time', ASCENDING)])
+        return utc_timestamp_to_datetime(document['next_run_time']) if document else None
+
+    def get_all_jobs(self):
+        return self._get_jobs({})
+
+    def add_job(self, job):
+        try:
+            self.collection.insert({
+                '_id': job.id,
+                'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+                'job_state': Binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
+            })
+        except DuplicateKeyError:
+            raise ConflictingIdError(job.id)
+
+    def update_job(self, job):
+        changes = {
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': Binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
+        }
+        result = self.collection.update({'_id': job.id}, {'$set': changes})
+        if result and result['n'] == 0:
+            raise JobLookupError(id)
+
+    def remove_job(self, job_id):
+        result = self.collection.remove(job_id)
+        if result and result['n'] == 0:
+            raise JobLookupError(job_id)
+
+    def remove_all_jobs(self):
+        self.collection.remove()
+
+    def shutdown(self):
+        self.connection.disconnect()
+
+    def _reconstitute_job(self, job_state):
+        job_state = pickle.loads(job_state)
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _get_jobs(self, conditions):
+        jobs = []
+        failed_job_ids = []
+        for document in self.collection.find(conditions, ['_id', 'job_state'], sort=[('next_run_time', ASCENDING)]):
+            try:
+                jobs.append(self._reconstitute_job(document['job_state']))
+            except:
+                self._logger.exception('Unable to restore job "%s" -- removing it', document['_id'])
+                failed_job_ids.append(document['_id'])
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            self.collection.remove({'_id': {'$in': failed_job_ids}})
+
+        return jobs
+
+    def __repr__(self):
+        return '<%s (client=%s)>' % (self.__class__.__name__, self.connection)

lib/apscheduler/jobstores/mongodb_store.py

@@ -1,84 +0,0 @@
-"""
-Stores jobs in a MongoDB database.
-"""
-import logging
-
-from apscheduler.jobstores.base import JobStore
-from apscheduler.job import Job
-
-try:
-    import cPickle as pickle
-except ImportError:  # pragma: nocover
-    import pickle
-
-try:
-    from bson.binary import Binary
-    from pymongo.connection import Connection
-except ImportError:  # pragma: nocover
-    raise ImportError('MongoDBJobStore requires PyMongo installed')
-
-logger = logging.getLogger(__name__)
-
-
-class MongoDBJobStore(JobStore):
-    def __init__(self, database='apscheduler', collection='jobs',
-                 connection=None, pickle_protocol=pickle.HIGHEST_PROTOCOL,
-                 **connect_args):
-        self.jobs = []
-        self.pickle_protocol = pickle_protocol
-
-        if not database:
-            raise ValueError('The "database" parameter must not be empty')
-        if not collection:
-            raise ValueError('The "collection" parameter must not be empty')
-
-        if connection:
-            self.connection = connection
-        else:
-            self.connection = Connection(**connect_args)
-
-        self.collection = self.connection[database][collection]
-
-    def add_job(self, job):
-        job_dict = job.__getstate__()
-        job_dict['trigger'] = Binary(pickle.dumps(job.trigger,
-                                                  self.pickle_protocol))
-        job_dict['args'] = Binary(pickle.dumps(job.args,
-                                               self.pickle_protocol))
-        job_dict['kwargs'] = Binary(pickle.dumps(job.kwargs,
-                                                 self.pickle_protocol))
-        job.id = self.collection.insert(job_dict)
-        self.jobs.append(job)
-
-    def remove_job(self, job):
-        self.collection.remove(job.id)
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        jobs = []
-        for job_dict in self.collection.find():
-            try:
-                job = Job.__new__(Job)
-                job_dict['id'] = job_dict.pop('_id')
-                job_dict['trigger'] = pickle.loads(job_dict['trigger'])
-                job_dict['args'] = pickle.loads(job_dict['args'])
-                job_dict['kwargs'] = pickle.loads(job_dict['kwargs'])
-                job.__setstate__(job_dict)
-                jobs.append(job)
-            except Exception:
-                job_name = job_dict.get('name', '(unknown)')
-                logger.exception('Unable to restore job "%s"', job_name)
-        self.jobs = jobs
-
-    def update_job(self, job):
-        spec = {'_id': job.id}
-        document = {'$set': {'next_run_time': job.next_run_time},
-                    '$inc': {'runs': 1}}
-        self.collection.update(spec, document)
-
-    def close(self):
-        self.connection.disconnect()
-
-    def __repr__(self):
-        connection = self.collection.database.connection
-        return '<%s (connection=%s)>' % (self.__class__.__name__, connection)

lib/apscheduler/jobstores/ram_store.py

@@ -1,25 +0,0 @@
-"""
-Stores jobs in an array in RAM. Provides no persistence support.
-"""
-
-from apscheduler.jobstores.base import JobStore
-
-
-class RAMJobStore(JobStore):
-    def __init__(self):
-        self.jobs = []
-
-    def add_job(self, job):
-        self.jobs.append(job)
-
-    def update_job(self, job):
-        pass
-
-    def remove_job(self, job):
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        pass
-
-    def __repr__(self):
-        return '<%s>' % (self.__class__.__name__)

lib/apscheduler/jobstores/redis.py

@@ -0,0 +1,138 @@
+from __future__ import absolute_import
+
+import six
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from redis import StrictRedis
+except ImportError:  # pragma: nocover
+    raise ImportError('RedisJobStore requires redis installed')
+
+
+class RedisJobStore(BaseJobStore):
+    """
+    Stores jobs in a Redis database. Any leftover keyword arguments are directly passed to redis's StrictRedis.
+
+    Plugin alias: ``redis``
+
+    :param int db: the database number to store jobs in
+    :param str jobs_key: key to store jobs in
+    :param str run_times_key: key to store the jobs' run times in
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    """
+
+    def __init__(self, db=0, jobs_key='apscheduler.jobs', run_times_key='apscheduler.run_times',
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL, **connect_args):
+        super(RedisJobStore, self).__init__()
+
+        if db is None:
+            raise ValueError('The "db" parameter must not be empty')
+        if not jobs_key:
+            raise ValueError('The "jobs_key" parameter must not be empty')
+        if not run_times_key:
+            raise ValueError('The "run_times_key" parameter must not be empty')
+
+        self.pickle_protocol = pickle_protocol
+        self.jobs_key = jobs_key
+        self.run_times_key = run_times_key
+        self.redis = StrictRedis(db=int(db), **connect_args)
+
+    def lookup_job(self, job_id):
+        job_state = self.redis.hget(self.jobs_key, job_id)
+        return self._reconstitute_job(job_state) if job_state else None
+
+    def get_due_jobs(self, now):
+        timestamp = datetime_to_utc_timestamp(now)
+        job_ids = self.redis.zrangebyscore(self.run_times_key, 0, timestamp)
+        if job_ids:
+            job_states = self.redis.hmget(self.jobs_key, *job_ids)
+            return self._reconstitute_jobs(six.moves.zip(job_ids, job_states))
+        return []
+
+    def get_next_run_time(self):
+        next_run_time = self.redis.zrange(self.run_times_key, 0, 0, withscores=True)
+        if next_run_time:
+            return utc_timestamp_to_datetime(next_run_time[0][1])
+
+    def get_all_jobs(self):
+        job_states = self.redis.hgetall(self.jobs_key)
+        jobs = self._reconstitute_jobs(six.iteritems(job_states))
+        return sorted(jobs, key=lambda job: job.next_run_time)
+
+    def add_job(self, job):
+        if self.redis.hexists(self.jobs_key, job.id):
+            raise ConflictingIdError(job.id)
+
+        with self.redis.pipeline() as pipe:
+            pipe.multi()
+            pipe.hset(self.jobs_key, job.id, pickle.dumps(job.__getstate__(), self.pickle_protocol))
+            pipe.zadd(self.run_times_key, datetime_to_utc_timestamp(job.next_run_time), job.id)
+            pipe.execute()
+
+    def update_job(self, job):
+        if not self.redis.hexists(self.jobs_key, job.id):
+            raise JobLookupError(job.id)
+
+        with self.redis.pipeline() as pipe:
+            pipe.hset(self.jobs_key, job.id, pickle.dumps(job.__getstate__(), self.pickle_protocol))
+            if job.next_run_time:
+                pipe.zadd(self.run_times_key, datetime_to_utc_timestamp(job.next_run_time), job.id)
+            else:
+                pipe.zrem(self.run_times_key, job.id)
+            pipe.execute()
+
+    def remove_job(self, job_id):
+        if not self.redis.hexists(self.jobs_key, job_id):
+            raise JobLookupError(job_id)
+
+        with self.redis.pipeline() as pipe:
+            pipe.hdel(self.jobs_key, job_id)
+            pipe.zrem(self.run_times_key, job_id)
+            pipe.execute()
+
+    def remove_all_jobs(self):
+        with self.redis.pipeline() as pipe:
+            pipe.delete(self.jobs_key)
+            pipe.delete(self.run_times_key)
+            pipe.execute()
+
+    def shutdown(self):
+        self.redis.connection_pool.disconnect()
+
+    def _reconstitute_job(self, job_state):
+        job_state = pickle.loads(job_state)
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _reconstitute_jobs(self, job_states):
+        jobs = []
+        failed_job_ids = []
+        for job_id, job_state in job_states:
+            try:
+                jobs.append(self._reconstitute_job(job_state))
+            except:
+                self._logger.exception('Unable to restore job "%s" -- removing it', job_id)
+                failed_job_ids.append(job_id)
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            with self.redis.pipeline() as pipe:
+                pipe.hdel(self.jobs_key, *failed_job_ids)
+                pipe.zrem(self.run_times_key, *failed_job_ids)
+                pipe.execute()
+
+        return jobs
+
+    def __repr__(self):
+        return '<%s>' % self.__class__.__name__

lib/apscheduler/jobstores/shelve_store.py

@@ -1,65 +0,0 @@
-"""
-Stores jobs in a file governed by the :mod:`shelve` module.
-"""
-
-import shelve
-import pickle
-import random
-import logging
-
-from apscheduler.jobstores.base import JobStore
-from apscheduler.job import Job
-from apscheduler.util import itervalues
-
-logger = logging.getLogger(__name__)
-
-
-class ShelveJobStore(JobStore):
-    MAX_ID = 1000000
-
-    def __init__(self, path, pickle_protocol=pickle.HIGHEST_PROTOCOL):
-        self.jobs = []
-        self.path = path
-        self.pickle_protocol = pickle_protocol
-        self.store = shelve.open(path, 'c', self.pickle_protocol)
-
-    def _generate_id(self):
-        id = None
-        while not id:
-            id = str(random.randint(1, self.MAX_ID))
-            if not id in self.store:
-                return id
-
-    def add_job(self, job):
-        job.id = self._generate_id()
-        self.jobs.append(job)
-        self.store[job.id] = job.__getstate__()
-
-    def update_job(self, job):
-        job_dict = self.store[job.id]
-        job_dict['next_run_time'] = job.next_run_time
-        job_dict['runs'] = job.runs
-        self.store[job.id] = job_dict
-
-    def remove_job(self, job):
-        del self.store[job.id]
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        jobs = []
-        for job_dict in itervalues(self.store):
-            try:
-                job = Job.__new__(Job)
-                job.__setstate__(job_dict)
-                jobs.append(job)
-            except Exception:
-                job_name = job_dict.get('name', '(unknown)')
-                logger.exception('Unable to restore job "%s"', job_name)
-
-        self.jobs = jobs
-
-    def close(self):
-        self.store.close()
-
-    def __repr__(self):
-        return '<%s (path=%s)>' % (self.__class__.__name__, self.path)

lib/apscheduler/jobstores/sqlalchemy.py

@@ -0,0 +1,137 @@
+from __future__ import absolute_import
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import maybe_ref, datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from sqlalchemy import create_engine, Table, Column, MetaData, Unicode, Float, LargeBinary, select
+    from sqlalchemy.exc import IntegrityError
+except ImportError:  # pragma: nocover
+    raise ImportError('SQLAlchemyJobStore requires SQLAlchemy installed')
+
+
+class SQLAlchemyJobStore(BaseJobStore):
+    """
+    Stores jobs in a database table using SQLAlchemy. The table will be created if it doesn't exist in the database.
+
+    Plugin alias: ``sqlalchemy``
+
+    :param str url: connection string (see `SQLAlchemy documentation
+                    <http://docs.sqlalchemy.org/en/latest/core/engines.html?highlight=create_engine#database-urls>`_
+                    on this)
+    :param engine: an SQLAlchemy Engine to use instead of creating a new one based on ``url``
+    :param str tablename: name of the table to store jobs in
+    :param metadata: a :class:`~sqlalchemy.MetaData` instance to use instead of creating a new one
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    """
+
+    def __init__(self, url=None, engine=None, tablename='apscheduler_jobs', metadata=None,
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL):
+        super(SQLAlchemyJobStore, self).__init__()
+        self.pickle_protocol = pickle_protocol
+        metadata = maybe_ref(metadata) or MetaData()
+
+        if engine:
+            self.engine = maybe_ref(engine)
+        elif url:
+            self.engine = create_engine(url)
+        else:
+            raise ValueError('Need either "engine" or "url" defined')
+
+        # 191 = max key length in MySQL for InnoDB/utf8mb4 tables, 25 = precision that translates to an 8-byte float
+        self.jobs_t = Table(
+            tablename, metadata,
+            Column('id', Unicode(191, _warn_on_bytestring=False), primary_key=True),
+            Column('next_run_time', Float(25), index=True),
+            Column('job_state', LargeBinary, nullable=False)
+        )
+
+        self.jobs_t.create(self.engine, True)
+
+    def lookup_job(self, job_id):
+        selectable = select([self.jobs_t.c.job_state]).where(self.jobs_t.c.id == job_id)
+        job_state = self.engine.execute(selectable).scalar()
+        return self._reconstitute_job(job_state) if job_state else None
+
+    def get_due_jobs(self, now):
+        timestamp = datetime_to_utc_timestamp(now)
+        return self._get_jobs(self.jobs_t.c.next_run_time <= timestamp)
+
+    def get_next_run_time(self):
+        selectable = select([self.jobs_t.c.next_run_time]).where(self.jobs_t.c.next_run_time != None).\
+            order_by(self.jobs_t.c.next_run_time).limit(1)
+        next_run_time = self.engine.execute(selectable).scalar()
+        return utc_timestamp_to_datetime(next_run_time)
+
+    def get_all_jobs(self):
+        return self._get_jobs()
+
+    def add_job(self, job):
+        insert = self.jobs_t.insert().values(**{
+            'id': job.id,
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': pickle.dumps(job.__getstate__(), self.pickle_protocol)
+        })
+        try:
+            self.engine.execute(insert)
+        except IntegrityError:
+            raise ConflictingIdError(job.id)
+
+    def update_job(self, job):
+        update = self.jobs_t.update().values(**{
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': pickle.dumps(job.__getstate__(), self.pickle_protocol)
+        }).where(self.jobs_t.c.id == job.id)
+        result = self.engine.execute(update)
+        if result.rowcount == 0:
+            raise JobLookupError(id)
+
+    def remove_job(self, job_id):
+        delete = self.jobs_t.delete().where(self.jobs_t.c.id == job_id)
+        result = self.engine.execute(delete)
+        if result.rowcount == 0:
+            raise JobLookupError(job_id)
+
+    def remove_all_jobs(self):
+        delete = self.jobs_t.delete()
+        self.engine.execute(delete)
+
+    def shutdown(self):
+        self.engine.dispose()
+
+    def _reconstitute_job(self, job_state):
+        job_state = pickle.loads(job_state)
+        job_state['jobstore'] = self
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _get_jobs(self, *conditions):
+        jobs = []
+        selectable = select([self.jobs_t.c.id, self.jobs_t.c.job_state]).order_by(self.jobs_t.c.next_run_time)
+        selectable = selectable.where(*conditions) if conditions else selectable
+        failed_job_ids = set()
+        for row in self.engine.execute(selectable):
+            try:
+                jobs.append(self._reconstitute_job(row.job_state))
+            except:
+                self._logger.exception('Unable to restore job "%s" -- removing it', row.id)
+                failed_job_ids.add(row.id)
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            delete = self.jobs_t.delete().where(self.jobs_t.c.id.in_(failed_job_ids))
+            self.engine.execute(delete)
+
+        return jobs
+
+    def __repr__(self):
+        return '<%s (url=%s)>' % (self.__class__.__name__, self.engine.url)

lib/apscheduler/jobstores/sqlalchemy_store.py

@@ -1,87 +0,0 @@
-"""
-Stores jobs in a database table using SQLAlchemy.
-"""
-import pickle
-import logging
-
-from apscheduler.jobstores.base import JobStore
-from apscheduler.job import Job
-
-try:
-    from sqlalchemy import *
-except ImportError:  # pragma: nocover
-    raise ImportError('SQLAlchemyJobStore requires SQLAlchemy installed')
-
-logger = logging.getLogger(__name__)
-
-
-class SQLAlchemyJobStore(JobStore):
-    def __init__(self, url=None, engine=None, tablename='apscheduler_jobs',
-                 metadata=None, pickle_protocol=pickle.HIGHEST_PROTOCOL):
-        self.jobs = []
-        self.pickle_protocol = pickle_protocol
-
-        if engine:
-            self.engine = engine
-        elif url:
-            self.engine = create_engine(url)
-        else:
-            raise ValueError('Need either "engine" or "url" defined')
-
-        self.jobs_t = Table(tablename, metadata or MetaData(),
-            Column('id', Integer,
-                   Sequence(tablename + '_id_seq', optional=True),
-                   primary_key=True),
-            Column('trigger', PickleType(pickle_protocol, mutable=False),
-                   nullable=False),
-            Column('func_ref', String(1024), nullable=False),
-            Column('args', PickleType(pickle_protocol, mutable=False),
-                   nullable=False),
-            Column('kwargs', PickleType(pickle_protocol, mutable=False),
-                   nullable=False),
-            Column('name', Unicode(1024), unique=True),
-            Column('misfire_grace_time', Integer, nullable=False),
-            Column('coalesce', Boolean, nullable=False),
-            Column('max_runs', Integer),
-            Column('max_instances', Integer),
-            Column('next_run_time', DateTime, nullable=False),
-            Column('runs', BigInteger))
-
-        self.jobs_t.create(self.engine, True)
-
-    def add_job(self, job):
-        job_dict = job.__getstate__()
-        result = self.engine.execute(self.jobs_t.insert().values(**job_dict))
-        job.id = result.inserted_primary_key[0]
-        self.jobs.append(job)
-
-    def remove_job(self, job):
-        delete = self.jobs_t.delete().where(self.jobs_t.c.id == job.id)
-        self.engine.execute(delete)
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        jobs = []
-        for row in self.engine.execute(select([self.jobs_t])):
-            try:
-                job = Job.__new__(Job)
-                job_dict = dict(row.items())
-                job.__setstate__(job_dict)
-                jobs.append(job)
-            except Exception:
-                job_name = job_dict.get('name', '(unknown)')
-                logger.exception('Unable to restore job "%s"', job_name)
-        self.jobs = jobs
-
-    def update_job(self, job):
-        job_dict = job.__getstate__()
-        update = self.jobs_t.update().where(self.jobs_t.c.id == job.id).\
-            values(next_run_time=job_dict['next_run_time'],
-                   runs=job_dict['runs'])
-        self.engine.execute(update)
-
-    def close(self):
-        self.engine.dispose()
-
-    def __repr__(self):
-        return '<%s (url=%s)>' % (self.__class__.__name__, self.engine.url)

lib/apscheduler/scheduler.py

@@ -1,559 +0,0 @@
-"""
-This module is the main part of the library. It houses the Scheduler class
-and related exceptions.
-"""
-
-from threading import Thread, Event, Lock
-from datetime import datetime, timedelta
-from logging import getLogger
-import os
-import sys
-
-from apscheduler.util import *
-from apscheduler.triggers import SimpleTrigger, IntervalTrigger, CronTrigger
-from apscheduler.jobstores.ram_store import RAMJobStore
-from apscheduler.job import Job, MaxInstancesReachedError
-from apscheduler.events import *
-from apscheduler.threadpool import ThreadPool
-
-logger = getLogger(__name__)
-
-
-class SchedulerAlreadyRunningError(Exception):
-    """
-    Raised when attempting to start or configure the scheduler when it's
-    already running.
-    """
-
-    def __str__(self):
-        return 'Scheduler is already running'
-
-
-class Scheduler(object):
-    """
-    This class is responsible for scheduling jobs and triggering
-    their execution.
-    """
-
-    _stopped = False
-    _thread = None
-
-    def __init__(self, gconfig={}, **options):
-        self._wakeup = Event()
-        self._jobstores = {}
-        self._jobstores_lock = Lock()
-        self._listeners = []
-        self._listeners_lock = Lock()
-        self._pending_jobs = []
-        self.configure(gconfig, **options)
-
-    def configure(self, gconfig={}, **options):
-        """
-        Reconfigures the scheduler with the given options. Can only be done
-        when the scheduler isn't running.
-        """
-        if self.running:
-            raise SchedulerAlreadyRunningError
-
-        # Set general options
-        config = combine_opts(gconfig, 'apscheduler.', options)
-        self.misfire_grace_time = int(config.pop('misfire_grace_time', 1))
-        self.coalesce = asbool(config.pop('coalesce', True))
-        self.daemonic = asbool(config.pop('daemonic', True))
-
-        # Configure the thread pool
-        if 'threadpool' in config:
-            self._threadpool = maybe_ref(config['threadpool'])
-        else:
-            threadpool_opts = combine_opts(config, 'threadpool.')
-            self._threadpool = ThreadPool(**threadpool_opts)
-
-        # Configure job stores
-        jobstore_opts = combine_opts(config, 'jobstore.')
-        jobstores = {}
-        for key, value in jobstore_opts.items():
-            store_name, option = key.split('.', 1)
-            opts_dict = jobstores.setdefault(store_name, {})
-            opts_dict[option] = value
-
-        for alias, opts in jobstores.items():
-            classname = opts.pop('class')
-            cls = maybe_ref(classname)
-            jobstore = cls(**opts)
-            self.add_jobstore(jobstore, alias, True)
-
-    def start(self):
-        """
-        Starts the scheduler in a new thread.
-        """
-        if self.running:
-            raise SchedulerAlreadyRunningError
-
-        # Create a RAMJobStore as the default if there is no default job store
-        if not 'default' in self._jobstores:
-            self.add_jobstore(RAMJobStore(), 'default', True)
-
-        # Schedule all pending jobs
-        for job, jobstore in self._pending_jobs:
-            self._real_add_job(job, jobstore, False)
-        del self._pending_jobs[:]
-
-        self._stopped = False
-        self._thread = Thread(target=self._main_loop, name='APScheduler')
-        self._thread.setDaemon(self.daemonic)
-        self._thread.start()
-
-    def shutdown(self, wait=True, shutdown_threadpool=True):
-        """
-        Shuts down the scheduler and terminates the thread.
-        Does not interrupt any currently running jobs.
-
-        :param wait: ``True`` to wait until all currently executing jobs have
-                     finished (if ``shutdown_threadpool`` is also ``True``)
-        :param shutdown_threadpool: ``True`` to shut down the thread pool
-        """
-        if not self.running:
-            return
-
-        self._stopped = True
-        self._wakeup.set()
-
-        # Shut down the thread pool
-        if shutdown_threadpool:
-            self._threadpool.shutdown(wait)
-
-        # Wait until the scheduler thread terminates
-        self._thread.join()
-
-    @property
-    def running(self):
-        return not self._stopped and self._thread and self._thread.isAlive()
-
-    def add_jobstore(self, jobstore, alias, quiet=False):
-        """
-        Adds a job store to this scheduler.
-
-        :param jobstore: job store to be added
-        :param alias: alias for the job store
-        :param quiet: True to suppress scheduler thread wakeup
-        :type jobstore: instance of
-            :class:`~apscheduler.jobstores.base.JobStore`
-        :type alias: str
-        """
-        self._jobstores_lock.acquire()
-        try:
-            if alias in self._jobstores:
-                raise KeyError('Alias "%s" is already in use' % alias)
-            self._jobstores[alias] = jobstore
-            jobstore.load_jobs()
-        finally:
-            self._jobstores_lock.release()
-
-        # Notify listeners that a new job store has been added
-        self._notify_listeners(JobStoreEvent(EVENT_JOBSTORE_ADDED, alias))
-
-        # Notify the scheduler so it can scan the new job store for jobs
-        if not quiet:
-            self._wakeup.set()
-
-    def remove_jobstore(self, alias):
-        """
-        Removes the job store by the given alias from this scheduler.
-
-        :type alias: str
-        """
-        self._jobstores_lock.acquire()
-        try:
-            try:
-                del self._jobstores[alias]
-            except KeyError:
-                raise KeyError('No such job store: %s' % alias)
-        finally:
-            self._jobstores_lock.release()
-
-        # Notify listeners that a job store has been removed
-        self._notify_listeners(JobStoreEvent(EVENT_JOBSTORE_REMOVED, alias))
-
-    def add_listener(self, callback, mask=EVENT_ALL):
-        """
-        Adds a listener for scheduler events. When a matching event occurs,
-        ``callback`` is executed with the event object as its sole argument.
-        If the ``mask`` parameter is not provided, the callback will receive
-        events of all types.
-
-        :param callback: any callable that takes one argument
-        :param mask: bitmask that indicates which events should be listened to
-        """
-        self._listeners_lock.acquire()
-        try:
-            self._listeners.append((callback, mask))
-        finally:
-            self._listeners_lock.release()
-
-    def remove_listener(self, callback):
-        """
-        Removes a previously added event listener.
-        """
-        self._listeners_lock.acquire()
-        try:
-            for i, (cb, _) in enumerate(self._listeners):
-                if callback == cb:
-                    del self._listeners[i]
-        finally:
-            self._listeners_lock.release()
-
-    def _notify_listeners(self, event):
-        self._listeners_lock.acquire()
-        try:
-            listeners = tuple(self._listeners)
-        finally:
-            self._listeners_lock.release()
-
-        for cb, mask in listeners:
-            if event.code & mask:
-                try:
-                    cb(event)
-                except:
-                    logger.exception('Error notifying listener')
-
-    def _real_add_job(self, job, jobstore, wakeup):
-        job.compute_next_run_time(datetime.now())
-        if not job.next_run_time:
-            raise ValueError('Not adding job since it would never be run')
-
-        self._jobstores_lock.acquire()
-        try:
-            try:
-                store = self._jobstores[jobstore]
-            except KeyError:
-                raise KeyError('No such job store: %s' % jobstore)
-            store.add_job(job)
-        finally:
-            self._jobstores_lock.release()
-
-        # Notify listeners that a new job has been added
-        event = JobStoreEvent(EVENT_JOBSTORE_JOB_ADDED, jobstore, job)
-        self._notify_listeners(event)
-
-        logger.info('Added job "%s" to job store "%s"', job, jobstore)
-
-        # Notify the scheduler about the new job
-        if wakeup:
-            self._wakeup.set()
-
-    def add_job(self, trigger, func, args, kwargs, jobstore='default',
-                **options):
-        """
-        Adds the given job to the job list and notifies the scheduler thread.
-
-        :param trigger: alias of the job store to store the job in
-        :param func: callable to run at the given time
-        :param args: list of positional arguments to call func with
-        :param kwargs: dict of keyword arguments to call func with
-        :param jobstore: alias of the job store to store the job in
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        job = Job(trigger, func, args or [], kwargs or {},
-                  options.pop('misfire_grace_time', self.misfire_grace_time),
-                  options.pop('coalesce', self.coalesce), **options)
-        if not self.running:
-            self._pending_jobs.append((job, jobstore))
-            logger.info('Adding job tentatively -- it will be properly '
-                        'scheduled when the scheduler starts')
-        else:
-            self._real_add_job(job, jobstore, True)
-        return job
-
-    def _remove_job(self, job, alias, jobstore):
-        jobstore.remove_job(job)
-
-        # Notify listeners that a job has been removed
-        event = JobStoreEvent(EVENT_JOBSTORE_JOB_REMOVED, alias, job)
-        self._notify_listeners(event)
-
-        logger.info('Removed job "%s"', job)
-
-    def add_date_job(self, func, date, args=None, kwargs=None, **options):
-        """
-        Schedules a job to be completed on a specific date and time.
-
-        :param func: callable to run at the given time
-        :param date: the date/time to run the job at
-        :param name: name of the job
-        :param jobstore: stored the job in the named (or given) job store
-        :param misfire_grace_time: seconds after the designated run time that
-            the job is still allowed to be run
-        :type date: :class:`datetime.date`
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        trigger = SimpleTrigger(date)
-        return self.add_job(trigger, func, args, kwargs, **options)
-
-    def add_interval_job(self, func, weeks=0, days=0, hours=0, minutes=0,
-                         seconds=0, start_date=None, args=None, kwargs=None,
-                         **options):
-        """
-        Schedules a job to be completed on specified intervals.
-
-        :param func: callable to run
-        :param weeks: number of weeks to wait
-        :param days: number of days to wait
-        :param hours: number of hours to wait
-        :param minutes: number of minutes to wait
-        :param seconds: number of seconds to wait
-        :param start_date: when to first execute the job and start the
-            counter (default is after the given interval)
-        :param args: list of positional arguments to call func with
-        :param kwargs: dict of keyword arguments to call func with
-        :param name: name of the job
-        :param jobstore: alias of the job store to add the job to
-        :param misfire_grace_time: seconds after the designated run time that
-            the job is still allowed to be run
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        interval = timedelta(weeks=weeks, days=days, hours=hours,
-                             minutes=minutes, seconds=seconds)
-        trigger = IntervalTrigger(interval, start_date)
-        return self.add_job(trigger, func, args, kwargs, **options)
-
-    def add_cron_job(self, func, year='*', month='*', day='*', week='*',
-                     day_of_week='*', hour='*', minute='*', second='*',
-                     start_date=None, args=None, kwargs=None, **options):
-        """
-        Schedules a job to be completed on times that match the given
-        expressions.
-
-        :param func: callable to run
-        :param year: year to run on
-        :param month: month to run on (0 = January)
-        :param day: day of month to run on
-        :param week: week of the year to run on
-        :param day_of_week: weekday to run on (0 = Monday)
-        :param hour: hour to run on
-        :param second: second to run on
-        :param args: list of positional arguments to call func with
-        :param kwargs: dict of keyword arguments to call func with
-        :param name: name of the job
-        :param jobstore: alias of the job store to add the job to
-        :param misfire_grace_time: seconds after the designated run time that
-            the job is still allowed to be run
-        :return: the scheduled job
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        trigger = CronTrigger(year=year, month=month, day=day, week=week,
-                              day_of_week=day_of_week, hour=hour,
-                              minute=minute, second=second,
-                              start_date=start_date)
-        return self.add_job(trigger, func, args, kwargs, **options)
-
-    def cron_schedule(self, **options):
-        """
-        Decorator version of :meth:`add_cron_job`.
-        This decorator does not wrap its host function.
-        Unscheduling decorated functions is possible by passing the ``job``
-        attribute of the scheduled function to :meth:`unschedule_job`.
-        """
-        def inner(func):
-            func.job = self.add_cron_job(func, **options)
-            return func
-        return inner
-
-    def interval_schedule(self, **options):
-        """
-        Decorator version of :meth:`add_interval_job`.
-        This decorator does not wrap its host function.
-        Unscheduling decorated functions is possible by passing the ``job``
-        attribute of the scheduled function to :meth:`unschedule_job`.
-        """
-        def inner(func):
-            func.job = self.add_interval_job(func, **options)
-            return func
-        return inner
-
-    def get_jobs(self):
-        """
-        Returns a list of all scheduled jobs.
-
-        :return: list of :class:`~apscheduler.job.Job` objects
-        """
-        self._jobstores_lock.acquire()
-        try:
-            jobs = []
-            for jobstore in itervalues(self._jobstores):
-                jobs.extend(jobstore.jobs)
-            return jobs
-        finally:
-            self._jobstores_lock.release()
-
-    def unschedule_job(self, job):
-        """
-        Removes a job, preventing it from being run any more.
-        """
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                if job in list(jobstore.jobs):
-                    self._remove_job(job, alias, jobstore)
-                    return
-        finally:
-            self._jobstores_lock.release()
-
-        raise KeyError('Job "%s" is not scheduled in any job store' % job)
-
-    def unschedule_func(self, func):
-        """
-        Removes all jobs that would execute the given function.
-        """
-        found = False
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                for job in list(jobstore.jobs):
-                    if job.func == func:
-                        self._remove_job(job, alias, jobstore)
-                        found = True
-        finally:
-            self._jobstores_lock.release()
-
-        if not found:
-            raise KeyError('The given function is not scheduled in this '
-                           'scheduler')
-
-    def print_jobs(self, out=None):
-        """
-        Prints out a textual listing of all jobs currently scheduled on this
-        scheduler.
-
-        :param out: a file-like object to print to (defaults to **sys.stdout**
-                    if nothing is given)
-        """
-        out = out or sys.stdout
-        job_strs = []
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                job_strs.append('Jobstore %s:' % alias)
-                if jobstore.jobs:
-                    for job in jobstore.jobs:
-                        job_strs.append('    %s' % job)
-                else:
-                    job_strs.append('    No scheduled jobs')
-        finally:
-            self._jobstores_lock.release()
-
-        out.write(os.linesep.join(job_strs))
-
-    def _run_job(self, job, run_times):
-        """
-        Acts as a harness that runs the actual job code in a thread.
-        """
-        for run_time in run_times:
-            # See if the job missed its run time window, and handle possible
-            # misfires accordingly
-            difference = datetime.now() - run_time
-            grace_time = timedelta(seconds=job.misfire_grace_time)
-            if difference > grace_time:
-                # Notify listeners about a missed run
-                event = JobEvent(EVENT_JOB_MISSED, job, run_time)
-                self._notify_listeners(event)
-                logger.warning('Run time of job "%s" was missed by %s',
-                               job, difference)
-            else:
-                try:
-                    job.add_instance()
-                except MaxInstancesReachedError:
-                    event = JobEvent(EVENT_JOB_MISSED, job, run_time)
-                    self._notify_listeners(event)
-                    logger.warning('Execution of job "%s" skipped: '
-                                   'maximum number of running instances '
-                                   'reached (%d)', job, job.max_instances)
-                    break
-
-                logger.info('Running job "%s" (scheduled at %s)', job,
-                            run_time)
-
-                try:
-                    retval = job.func(*job.args, **job.kwargs)
-                except:
-                    # Notify listeners about the exception
-                    exc, tb = sys.exc_info()[1:]
-                    event = JobEvent(EVENT_JOB_ERROR, job, run_time,
-                                     exception=exc, traceback=tb)
-                    self._notify_listeners(event)
-
-                    logger.exception('Job "%s" raised an exception', job)
-                else:
-                    # Notify listeners about successful execution
-                    event = JobEvent(EVENT_JOB_EXECUTED, job, run_time,
-                                     retval=retval)
-                    self._notify_listeners(event)
-
-                    logger.info('Job "%s" executed successfully', job)
-
-                job.remove_instance()
-
-                # If coalescing is enabled, don't attempt any further runs
-                if job.coalesce:
-                    break
-
-    def _process_jobs(self, now):
-        """
-        Iterates through jobs in every jobstore, starts pending jobs
-        and figures out the next wakeup time.
-        """
-        next_wakeup_time = None
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                for job in tuple(jobstore.jobs):
-                    run_times = job.get_run_times(now)
-                    if run_times:
-                        self._threadpool.submit(self._run_job, job, run_times)
-
-                        # Increase the job's run count
-                        if job.coalesce:
-                            job.runs += 1
-                        else:
-                            job.runs += len(run_times)
-
-                        # Update the job, but don't keep finished jobs around
-                        if job.compute_next_run_time(now + timedelta(microseconds=1)):
-                            jobstore.update_job(job)
-                        else:
-                            self._remove_job(job, alias, jobstore)
-
-                    if not next_wakeup_time:
-                        next_wakeup_time = job.next_run_time
-                    elif job.next_run_time:
-                        next_wakeup_time = min(next_wakeup_time,
-                                               job.next_run_time)
-            return next_wakeup_time
-        finally:
-            self._jobstores_lock.release()
-
-    def _main_loop(self):
-        """Executes jobs on schedule."""
-
-        logger.info('Scheduler started')
-        self._notify_listeners(SchedulerEvent(EVENT_SCHEDULER_START))
-
-        self._wakeup.clear()
-        while not self._stopped:
-            logger.debug('Looking for jobs to run')
-            now = datetime.now()
-            next_wakeup_time = self._process_jobs(now)
-
-            # Sleep until the next job is scheduled to be run,
-            # a new job is added or the scheduler is stopped
-            if next_wakeup_time is not None:
-                wait_seconds = time_difference(next_wakeup_time, now)
-                logger.debug('Next wakeup is due at %s (in %f seconds)',
-                             next_wakeup_time, wait_seconds)
-                self._wakeup.wait(wait_seconds)
-            else:
-                logger.debug('No jobs; waiting until a job is added')
-                self._wakeup.wait()
-            self._wakeup.clear()
-
-        logger.info('Scheduler has been shut down')
-        self._notify_listeners(SchedulerEvent(EVENT_SCHEDULER_SHUTDOWN))

lib/apscheduler/schedulers/__init__.py

@@ -0,0 +1,12 @@
+class SchedulerAlreadyRunningError(Exception):
+    """Raised when attempting to start or configure the scheduler when it's already running."""
+
+    def __str__(self):
+        return 'Scheduler is already running'
+
+
+class SchedulerNotRunningError(Exception):
+    """Raised when attempting to shutdown the scheduler when it's not running."""
+
+    def __str__(self):
+        return 'Scheduler is not running'

lib/apscheduler/schedulers/asyncio.py

@@ -0,0 +1,68 @@
+from __future__ import absolute_import
+from functools import wraps
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.util import maybe_ref
+
+try:
+    import asyncio
+except ImportError:  # pragma: nocover
+    try:
+        import trollius as asyncio
+    except ImportError:
+        raise ImportError('AsyncIOScheduler requires either Python 3.4 or the asyncio package installed')
+
+
+def run_in_event_loop(func):
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        self._eventloop.call_soon_threadsafe(func, self, *args, **kwargs)
+    return wrapper
+
+
+class AsyncIOScheduler(BaseScheduler):
+    """
+    A scheduler that runs on an asyncio (:pep:`3156`) event loop.
+
+    Extra options:
+
+    ============== =============================================================
+    ``event_loop`` AsyncIO event loop to use (defaults to the global event loop)
+    ============== =============================================================
+    """
+
+    _eventloop = None
+    _timeout = None
+
+    def start(self):
+        super(AsyncIOScheduler, self).start()
+        self.wakeup()
+
+    @run_in_event_loop
+    def shutdown(self, wait=True):
+        super(AsyncIOScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _configure(self, config):
+        self._eventloop = maybe_ref(config.pop('event_loop', None)) or asyncio.get_event_loop()
+        super(AsyncIOScheduler, self)._configure(config)
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._timeout = self._eventloop.call_later(wait_seconds, self.wakeup)
+
+    def _stop_timer(self):
+        if self._timeout:
+            self._timeout.cancel()
+            del self._timeout
+
+    @run_in_event_loop
+    def wakeup(self):
+        self._stop_timer()
+        wait_seconds = self._process_jobs()
+        self._start_timer(wait_seconds)
+
+    def _create_default_executor(self):
+        from apscheduler.executors.asyncio import AsyncIOExecutor
+        return AsyncIOExecutor()

lib/apscheduler/schedulers/background.py

@@ -0,0 +1,39 @@
+from __future__ import absolute_import
+from threading import Thread, Event
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.schedulers.blocking import BlockingScheduler
+from apscheduler.util import asbool
+
+
+class BackgroundScheduler(BlockingScheduler):
+    """
+    A scheduler that runs in the background using a separate thread
+    (:meth:`~apscheduler.schedulers.base.BaseScheduler.start` will return immediately).
+
+    Extra options:
+
+    ========== ============================================================================================
+    ``daemon`` Set the ``daemon`` option in the background thread (defaults to ``True``,
+               see `the documentation <https://docs.python.org/3.4/library/threading.html#thread-objects>`_
+               for further details)
+    ========== ============================================================================================
+    """
+
+    _thread = None
+
+    def _configure(self, config):
+        self._daemon = asbool(config.pop('daemon', True))
+        super(BackgroundScheduler, self)._configure(config)
+
+    def start(self):
+        BaseScheduler.start(self)
+        self._event = Event()
+        self._thread = Thread(target=self._main_loop, name='APScheduler')
+        self._thread.daemon = self._daemon
+        self._thread.start()
+
+    def shutdown(self, wait=True):
+        super(BackgroundScheduler, self).shutdown(wait)
+        self._thread.join()
+        del self._thread

lib/apscheduler/schedulers/base.py

@@ -0,0 +1,845 @@
+from __future__ import print_function
+from abc import ABCMeta, abstractmethod
+from collections import MutableMapping
+from threading import RLock
+from datetime import datetime
+from logging import getLogger
+import sys
+
+from pkg_resources import iter_entry_points
+from tzlocal import get_localzone
+import six
+
+from apscheduler.schedulers import SchedulerAlreadyRunningError, SchedulerNotRunningError
+from apscheduler.executors.base import MaxInstancesReachedError, BaseExecutor
+from apscheduler.executors.pool import ThreadPoolExecutor
+from apscheduler.jobstores.base import ConflictingIdError, JobLookupError, BaseJobStore
+from apscheduler.jobstores.memory import MemoryJobStore
+from apscheduler.job import Job
+from apscheduler.triggers.base import BaseTrigger
+from apscheduler.util import asbool, asint, astimezone, maybe_ref, timedelta_seconds, undefined
+from apscheduler.events import (
+    SchedulerEvent, JobEvent, EVENT_SCHEDULER_START, EVENT_SCHEDULER_SHUTDOWN, EVENT_JOBSTORE_ADDED,
+    EVENT_JOBSTORE_REMOVED, EVENT_ALL, EVENT_JOB_MODIFIED, EVENT_JOB_REMOVED, EVENT_JOB_ADDED, EVENT_EXECUTOR_ADDED,
+    EVENT_EXECUTOR_REMOVED, EVENT_ALL_JOBS_REMOVED)
+
+
+class BaseScheduler(six.with_metaclass(ABCMeta)):
+    """
+    Abstract base class for all schedulers. Takes the following keyword arguments:
+
+    :param str|logging.Logger logger: logger to use for the scheduler's logging (defaults to apscheduler.scheduler)
+    :param str|datetime.tzinfo timezone: the default time zone (defaults to the local timezone)
+    :param dict job_defaults: default values for newly added jobs
+    :param dict jobstores: a dictionary of job store alias -> job store instance or configuration dict
+    :param dict executors: a dictionary of executor alias -> executor instance or configuration dict
+
+    .. seealso:: :ref:`scheduler-config`
+    """
+
+    _trigger_plugins = dict((ep.name, ep) for ep in iter_entry_points('apscheduler.triggers'))
+    _trigger_classes = {}
+    _executor_plugins = dict((ep.name, ep) for ep in iter_entry_points('apscheduler.executors'))
+    _executor_classes = {}
+    _jobstore_plugins = dict((ep.name, ep) for ep in iter_entry_points('apscheduler.jobstores'))
+    _jobstore_classes = {}
+    _stopped = True
+
+    #
+    # Public API
+    #
+
+    def __init__(self, gconfig={}, **options):
+        super(BaseScheduler, self).__init__()
+        self._executors = {}
+        self._executors_lock = self._create_lock()
+        self._jobstores = {}
+        self._jobstores_lock = self._create_lock()
+        self._listeners = []
+        self._listeners_lock = self._create_lock()
+        self._pending_jobs = []
+        self.configure(gconfig, **options)
+
+    def configure(self, gconfig={}, prefix='apscheduler.', **options):
+        """
+        Reconfigures the scheduler with the given options. Can only be done when the scheduler isn't running.
+
+        :param dict gconfig: a "global" configuration dictionary whose values can be overridden by keyword arguments to
+                             this method
+        :param str|unicode prefix: pick only those keys from ``gconfig`` that are prefixed with this string
+                                   (pass an empty string or ``None`` to use all keys)
+        :raises SchedulerAlreadyRunningError: if the scheduler is already running
+        """
+
+        if self.running:
+            raise SchedulerAlreadyRunningError
+
+        # If a non-empty prefix was given, strip it from the keys in the global configuration dict
+        if prefix:
+            prefixlen = len(prefix)
+            gconfig = dict((key[prefixlen:], value) for key, value in six.iteritems(gconfig) if key.startswith(prefix))
+
+        # Create a structure from the dotted options (e.g. "a.b.c = d" -> {'a': {'b': {'c': 'd'}}})
+        config = {}
+        for key, value in six.iteritems(gconfig):
+            parts = key.split('.')
+            parent = config
+            key = parts.pop(0)
+            while parts:
+                parent = parent.setdefault(key, {})
+                key = parts.pop(0)
+            parent[key] = value
+
+        # Override any options with explicit keyword arguments
+        config.update(options)
+        self._configure(config)
+
+    @abstractmethod
+    def start(self):
+        """
+        Starts the scheduler. The details of this process depend on the implementation.
+
+        :raises SchedulerAlreadyRunningError: if the scheduler is already running
+        """
+
+        if self.running:
+            raise SchedulerAlreadyRunningError
+
+        with self._executors_lock:
+            # Create a default executor if nothing else is configured
+            if 'default' not in self._executors:
+                self.add_executor(self._create_default_executor(), 'default')
+
+            # Start all the executors
+            for alias, executor in six.iteritems(self._executors):
+                executor.start(self, alias)
+
+        with self._jobstores_lock:
+            # Create a default job store if nothing else is configured
+            if 'default' not in self._jobstores:
+                self.add_jobstore(self._create_default_jobstore(), 'default')
+
+            # Start all the job stores
+            for alias, store in six.iteritems(self._jobstores):
+                store.start(self, alias)
+
+            # Schedule all pending jobs
+            for job, jobstore_alias, replace_existing in self._pending_jobs:
+                self._real_add_job(job, jobstore_alias, replace_existing, False)
+            del self._pending_jobs[:]
+
+        self._stopped = False
+        self._logger.info('Scheduler started')
+
+        # Notify listeners that the scheduler has been started
+        self._dispatch_event(SchedulerEvent(EVENT_SCHEDULER_START))
+
+    @abstractmethod
+    def shutdown(self, wait=True):
+        """
+        Shuts down the scheduler. Does not interrupt any currently running jobs.
+
+        :param bool wait: ``True`` to wait until all currently executing jobs have finished
+        :raises SchedulerNotRunningError: if the scheduler has not been started yet
+        """
+
+        if not self.running:
+            raise SchedulerNotRunningError
+
+        self._stopped = True
+
+        # Shut down all executors
+        for executor in six.itervalues(self._executors):
+            executor.shutdown(wait)
+
+        # Shut down all job stores
+        for jobstore in six.itervalues(self._jobstores):
+            jobstore.shutdown()
+
+        self._logger.info('Scheduler has been shut down')
+        self._dispatch_event(SchedulerEvent(EVENT_SCHEDULER_SHUTDOWN))
+
+    @property
+    def running(self):
+        return not self._stopped
+
+    def add_executor(self, executor, alias='default', **executor_opts):
+        """
+        Adds an executor to this scheduler. Any extra keyword arguments will be passed to the executor plugin's
+        constructor, assuming that the first argument is the name of an executor plugin.
+
+        :param str|unicode|apscheduler.executors.base.BaseExecutor executor: either an executor instance or the name of
+            an executor plugin
+        :param str|unicode alias: alias for the scheduler
+        :raises ValueError: if there is already an executor by the given alias
+        """
+
+        with self._executors_lock:
+            if alias in self._executors:
+                raise ValueError('This scheduler already has an executor by the alias of "%s"' % alias)
+
+            if isinstance(executor, BaseExecutor):
+                self._executors[alias] = executor
+            elif isinstance(executor, six.string_types):
+                self._executors[alias] = executor = self._create_plugin_instance('executor', executor, executor_opts)
+            else:
+                raise TypeError('Expected an executor instance or a string, got %s instead' %
+                                executor.__class__.__name__)
+
+            # Start the executor right away if the scheduler is running
+            if self.running:
+                executor.start(self)
+
+        self._dispatch_event(SchedulerEvent(EVENT_EXECUTOR_ADDED, alias))
+
+    def remove_executor(self, alias, shutdown=True):
+        """
+        Removes the executor by the given alias from this scheduler.
+
+        :param str|unicode alias: alias of the executor
+        :param bool shutdown: ``True`` to shut down the executor after removing it
+        """
+
+        with self._jobstores_lock:
+            executor = self._lookup_executor(alias)
+            del self._executors[alias]
+
+        if shutdown:
+            executor.shutdown()
+
+        self._dispatch_event(SchedulerEvent(EVENT_EXECUTOR_REMOVED, alias))
+
+    def add_jobstore(self, jobstore, alias='default', **jobstore_opts):
+        """
+        Adds a job store to this scheduler. Any extra keyword arguments will be passed to the job store plugin's
+        constructor, assuming that the first argument is the name of a job store plugin.
+
+        :param str|unicode|apscheduler.jobstores.base.BaseJobStore jobstore: job store to be added
+        :param str|unicode alias: alias for the job store
+        :raises ValueError: if there is already a job store by the given alias
+        """
+
+        with self._jobstores_lock:
+            if alias in self._jobstores:
+                raise ValueError('This scheduler already has a job store by the alias of "%s"' % alias)
+
+            if isinstance(jobstore, BaseJobStore):
+                self._jobstores[alias] = jobstore
+            elif isinstance(jobstore, six.string_types):
+                self._jobstores[alias] = jobstore = self._create_plugin_instance('jobstore', jobstore, jobstore_opts)
+            else:
+                raise TypeError('Expected a job store instance or a string, got %s instead' %
+                                jobstore.__class__.__name__)
+
+            # Start the job store right away if the scheduler is running
+            if self.running:
+                jobstore.start(self, alias)
+
+        # Notify listeners that a new job store has been added
+        self._dispatch_event(SchedulerEvent(EVENT_JOBSTORE_ADDED, alias))
+
+        # Notify the scheduler so it can scan the new job store for jobs
+        if self.running:
+            self.wakeup()
+
+    def remove_jobstore(self, alias, shutdown=True):
+        """
+        Removes the job store by the given alias from this scheduler.
+
+        :param str|unicode alias: alias of the job store
+        :param bool shutdown: ``True`` to shut down the job store after removing it
+        """
+
+        with self._jobstores_lock:
+            jobstore = self._lookup_jobstore(alias)
+            del self._jobstores[alias]
+
+        if shutdown:
+            jobstore.shutdown()
+
+        self._dispatch_event(SchedulerEvent(EVENT_JOBSTORE_REMOVED, alias))
+
+    def add_listener(self, callback, mask=EVENT_ALL):
+        """
+        add_listener(callback, mask=EVENT_ALL)
+
+        Adds a listener for scheduler events. When a matching event occurs, ``callback`` is executed with the event
+        object as its sole argument. If the ``mask`` parameter is not provided, the callback will receive events of all
+        types.
+
+        :param callback: any callable that takes one argument
+        :param int mask: bitmask that indicates which events should be listened to
+
+        .. seealso:: :mod:`apscheduler.events`
+        .. seealso:: :ref:`scheduler-events`
+        """
+
+        with self._listeners_lock:
+            self._listeners.append((callback, mask))
+
+    def remove_listener(self, callback):
+        """Removes a previously added event listener."""
+
+        with self._listeners_lock:
+            for i, (cb, _) in enumerate(self._listeners):
+                if callback == cb:
+                    del self._listeners[i]
+
+    def add_job(self, func, trigger=None, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined,
+                coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default',
+                executor='default', replace_existing=False, **trigger_args):
+        """
+        add_job(func, trigger=None, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined, \
+            coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default', \
+            executor='default', replace_existing=False, **trigger_args)
+
+        Adds the given job to the job list and wakes up the scheduler if it's already running.
+
+        Any option that defaults to ``undefined`` will be replaced with the corresponding default value when the job is
+        scheduled (which happens when the scheduler is started, or immediately if the scheduler is already running).
+
+        The ``func`` argument can be given either as a callable object or a textual reference in the
+        ``package.module:some.object`` format, where the first half (separated by ``:``) is an importable module and the
+        second half is a reference to the callable object, relative to the module.
+
+        The ``trigger`` argument can either be:
+          #. the alias name of the trigger (e.g. ``date``, ``interval`` or ``cron``), in which case any extra keyword
+             arguments to this method are passed on to the trigger's constructor
+          #. an instance of a trigger class
+
+        :param func: callable (or a textual reference to one) to run at the given time
+        :param str|apscheduler.triggers.base.BaseTrigger trigger: trigger that determines when ``func`` is called
+        :param list|tuple args: list of positional arguments to call func with
+        :param dict kwargs: dict of keyword arguments to call func with
+        :param str|unicode id: explicit identifier for the job (for modifying it later)
+        :param str|unicode name: textual description of the job
+        :param int misfire_grace_time: seconds after the designated run time that the job is still allowed to be run
+        :param bool coalesce: run once instead of many times if the scheduler determines that the job should be run more
+                              than once in succession
+        :param int max_instances: maximum number of concurrently running instances allowed for this job
+        :param datetime next_run_time: when to first run the job, regardless of the trigger (pass ``None`` to add the
+                                       job as paused)
+        :param str|unicode jobstore: alias of the job store to store the job in
+        :param str|unicode executor: alias of the executor to run the job with
+        :param bool replace_existing: ``True`` to replace an existing job with the same ``id`` (but retain the
+                                      number of runs from the existing one)
+        :rtype: Job
+        """
+
+        job_kwargs = {
+            'trigger': self._create_trigger(trigger, trigger_args),
+            'executor': executor,
+            'func': func,
+            'args': tuple(args) if args is not None else (),
+            'kwargs': dict(kwargs) if kwargs is not None else {},
+            'id': id,
+            'name': name,
+            'misfire_grace_time': misfire_grace_time,
+            'coalesce': coalesce,
+            'max_instances': max_instances,
+            'next_run_time': next_run_time
+        }
+        job_kwargs = dict((key, value) for key, value in six.iteritems(job_kwargs) if value is not undefined)
+        job = Job(self, **job_kwargs)
+
+        # Don't really add jobs to job stores before the scheduler is up and running
+        with self._jobstores_lock:
+            if not self.running:
+                self._pending_jobs.append((job, jobstore, replace_existing))
+                self._logger.info('Adding job tentatively -- it will be properly scheduled when the scheduler starts')
+            else:
+                self._real_add_job(job, jobstore, replace_existing, True)
+
+        return job
+
+    def scheduled_job(self, trigger, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined,
+                      coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default',
+                      executor='default', **trigger_args):
+        """
+        scheduled_job(trigger, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined, \
+            coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default', \
+            executor='default',**trigger_args)
+
+        A decorator version of :meth:`add_job`, except that ``replace_existing`` is always ``True``.
+
+        .. important:: The ``id`` argument must be given if scheduling a job in a persistent job store. The scheduler
+           cannot, however, enforce this requirement.
+        """
+
+        def inner(func):
+            self.add_job(func, trigger, args, kwargs, id, name, misfire_grace_time, coalesce, max_instances,
+                         next_run_time, jobstore, executor, True, **trigger_args)
+            return func
+        return inner
+
+    def modify_job(self, job_id, jobstore=None, **changes):
+        """
+        Modifies the properties of a single job. Modifications are passed to this method as extra keyword arguments.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        """
+        with self._jobstores_lock:
+            job, jobstore = self._lookup_job(job_id, jobstore)
+            job._modify(**changes)
+            if jobstore:
+                self._lookup_jobstore(jobstore).update_job(job)
+
+        self._dispatch_event(JobEvent(EVENT_JOB_MODIFIED, job_id, jobstore))
+
+        # Wake up the scheduler since the job's next run time may have been changed
+        self.wakeup()
+
+    def reschedule_job(self, job_id, jobstore=None, trigger=None, **trigger_args):
+        """
+        Constructs a new trigger for a job and updates its next run time.
+        Extra keyword arguments are passed directly to the trigger's constructor.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        :param trigger: alias of the trigger type or a trigger instance
+        """
+
+        trigger = self._create_trigger(trigger, trigger_args)
+        now = datetime.now(self.timezone)
+        next_run_time = trigger.get_next_fire_time(None, now)
+        self.modify_job(job_id, jobstore, trigger=trigger, next_run_time=next_run_time)
+
+    def pause_job(self, job_id, jobstore=None):
+        """
+        Causes the given job not to be executed until it is explicitly resumed.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        """
+
+        self.modify_job(job_id, jobstore, next_run_time=None)
+
+    def resume_job(self, job_id, jobstore=None):
+        """
+        Resumes the schedule of the given job, or removes the job if its schedule is finished.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        """
+
+        with self._jobstores_lock:
+            job, jobstore = self._lookup_job(job_id, jobstore)
+            now = datetime.now(self.timezone)
+            next_run_time = job.trigger.get_next_fire_time(None, now)
+            if next_run_time:
+                self.modify_job(job_id, jobstore, next_run_time=next_run_time)
+            else:
+                self.remove_job(job.id, jobstore)
+
+    def get_jobs(self, jobstore=None, pending=None):
+        """
+        Returns a list of pending jobs (if the scheduler hasn't been started yet) and scheduled jobs, either from a
+        specific job store or from all of them.
+
+        :param str|unicode jobstore: alias of the job store
+        :param bool pending: ``False`` to leave out pending jobs (jobs that are waiting for the scheduler start to be
+                             added to their respective job stores), ``True`` to only include pending jobs, anything else
+                             to return both
+        :rtype: list[Job]
+        """
+
+        with self._jobstores_lock:
+            jobs = []
+
+            if pending is not False:
+                for job, alias, replace_existing in self._pending_jobs:
+                    if jobstore is None or alias == jobstore:
+                        jobs.append(job)
+
+            if pending is not True:
+                for alias, store in six.iteritems(self._jobstores):
+                    if jobstore is None or alias == jobstore:
+                        jobs.extend(store.get_all_jobs())
+
+            return jobs
+
+    def get_job(self, job_id, jobstore=None):
+        """
+        Returns the Job that matches the given ``job_id``.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that most likely contains the job
+        :return: the Job by the given ID, or ``None`` if it wasn't found
+        :rtype: Job
+        """
+
+        with self._jobstores_lock:
+            try:
+                return self._lookup_job(job_id, jobstore)[0]
+            except JobLookupError:
+                return
+
+    def remove_job(self, job_id, jobstore=None):
+        """
+        Removes a job, preventing it from being run any more.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        :raises JobLookupError: if the job was not found
+        """
+
+        with self._jobstores_lock:
+            # Check if the job is among the pending jobs
+            for i, (job, jobstore_alias, replace_existing) in enumerate(self._pending_jobs):
+                if job.id == job_id:
+                    del self._pending_jobs[i]
+                    jobstore = jobstore_alias
+                    break
+            else:
+                # Otherwise, try to remove it from each store until it succeeds or we run out of stores to check
+                for alias, store in six.iteritems(self._jobstores):
+                    if jobstore in (None, alias):
+                        try:
+                            store.remove_job(job_id)
+                        except JobLookupError:
+                            continue
+
+                        jobstore = alias
+                        break
+
+        if jobstore is None:
+            raise JobLookupError(job_id)
+
+        # Notify listeners that a job has been removed
+        event = JobEvent(EVENT_JOB_REMOVED, job_id, jobstore)
+        self._dispatch_event(event)
+
+        self._logger.info('Removed job %s', job_id)
+
+    def remove_all_jobs(self, jobstore=None):
+        """
+        Removes all jobs from the specified job store, or all job stores if none is given.
+
+        :param str|unicode jobstore: alias of the job store
+        """
+
+        with self._jobstores_lock:
+            if jobstore:
+                self._pending_jobs = [pending for pending in self._pending_jobs if pending[1] != jobstore]
+            else:
+                self._pending_jobs = []
+
+            for alias, store in six.iteritems(self._jobstores):
+                if jobstore in (None, alias):
+                    store.remove_all_jobs()
+
+        self._dispatch_event(SchedulerEvent(EVENT_ALL_JOBS_REMOVED, jobstore))
+
+    def print_jobs(self, jobstore=None, out=None):
+        """
+        print_jobs(jobstore=None, out=sys.stdout)
+
+        Prints out a textual listing of all jobs currently scheduled on either all job stores or just a specific one.
+
+        :param str|unicode jobstore: alias of the job store, ``None`` to list jobs from all stores
+        :param file out: a file-like object to print to (defaults to **sys.stdout** if nothing is given)
+        """
+
+        out = out or sys.stdout
+        with self._jobstores_lock:
+            if self._pending_jobs:
+                print(six.u('Pending jobs:'), file=out)
+                for job, jobstore_alias, replace_existing in self._pending_jobs:
+                    if jobstore in (None, jobstore_alias):
+                        print(six.u('    %s') % job, file=out)
+
+            for alias, store in six.iteritems(self._jobstores):
+                if jobstore in (None, alias):
+                    print(six.u('Jobstore %s:') % alias, file=out)
+                    jobs = store.get_all_jobs()
+                    if jobs:
+                        for job in jobs:
+                            print(six.u('    %s') % job, file=out)
+                    else:
+                        print(six.u('    No scheduled jobs'), file=out)
+
+    @abstractmethod
+    def wakeup(self):
+        """
+        Notifies the scheduler that there may be jobs due for execution.
+        Triggers :meth:`_process_jobs` to be run in an implementation specific manner.
+        """
+
+    #
+    # Private API
+    #
+
+    def _configure(self, config):
+        # Set general options
+        self._logger = maybe_ref(config.pop('logger', None)) or getLogger('apscheduler.scheduler')
+        self.timezone = astimezone(config.pop('timezone', None)) or get_localzone()
+
+        # Set the job defaults
+        job_defaults = config.get('job_defaults', {})
+        self._job_defaults = {
+            'misfire_grace_time': asint(job_defaults.get('misfire_grace_time', 1)),
+            'coalesce': asbool(job_defaults.get('coalesce', True)),
+            'max_instances': asint(job_defaults.get('max_instances', 1))
+        }
+
+        # Configure executors
+        self._executors.clear()
+        for alias, value in six.iteritems(config.get('executors', {})):
+            if isinstance(value, BaseExecutor):
+                self.add_executor(value, alias)
+            elif isinstance(value, MutableMapping):
+                executor_class = value.pop('class', None)
+                plugin = value.pop('type', None)
+                if plugin:
+                    executor = self._create_plugin_instance('executor', plugin, value)
+                elif executor_class:
+                    cls = maybe_ref(executor_class)
+                    executor = cls(**value)
+                else:
+                    raise ValueError('Cannot create executor "%s" -- either "type" or "class" must be defined' % alias)
+
+                self.add_executor(executor, alias)
+            else:
+                raise TypeError("Expected executor instance or dict for executors['%s'], got %s instead" % (
+                    alias, value.__class__.__name__))
+
+        # Configure job stores
+        self._jobstores.clear()
+        for alias, value in six.iteritems(config.get('jobstores', {})):
+            if isinstance(value, BaseJobStore):
+                self.add_jobstore(value, alias)
+            elif isinstance(value, MutableMapping):
+                jobstore_class = value.pop('class', None)
+                plugin = value.pop('type', None)
+                if plugin:
+                    jobstore = self._create_plugin_instance('jobstore', plugin, value)
+                elif jobstore_class:
+                    cls = maybe_ref(jobstore_class)
+                    jobstore = cls(**value)
+                else:
+                    raise ValueError('Cannot create job store "%s" -- either "type" or "class" must be defined' % alias)
+
+                self.add_jobstore(jobstore, alias)
+            else:
+                raise TypeError("Expected job store instance or dict for jobstores['%s'], got %s instead" % (
+                    alias, value.__class__.__name__))
+
+    def _create_default_executor(self):
+        """Creates a default executor store, specific to the particular scheduler type."""
+
+        return ThreadPoolExecutor()
+
+    def _create_default_jobstore(self):
+        """Creates a default job store, specific to the particular scheduler type."""
+
+        return MemoryJobStore()
+
+    def _lookup_executor(self, alias):
+        """
+        Returns the executor instance by the given name from the list of executors that were added to this scheduler.
+
+        :type alias: str
+        :raises KeyError: if no executor by the given alias is not found
+        """
+
+        try:
+            return self._executors[alias]
+        except KeyError:
+            raise KeyError('No such executor: %s' % alias)
+
+    def _lookup_jobstore(self, alias):
+        """
+        Returns the job store instance by the given name from the list of job stores that were added to this scheduler.
+
+        :type alias: str
+        :raises KeyError: if no job store by the given alias is not found
+        """
+
+        try:
+            return self._jobstores[alias]
+        except KeyError:
+            raise KeyError('No such job store: %s' % alias)
+
+    def _lookup_job(self, job_id, jobstore_alias):
+        """
+        Finds a job by its ID.
+
+        :type job_id: str
+        :param str jobstore_alias: alias of a job store to look in
+        :return tuple[Job, str]: a tuple of job, jobstore alias (jobstore alias is None in case of a pending job)
+        :raises JobLookupError: if no job by the given ID is found.
+        """
+
+        # Check if the job is among the pending jobs
+        for job, alias, replace_existing in self._pending_jobs:
+            if job.id == job_id:
+                return job, None
+
+        # Look in all job stores
+        for alias, store in six.iteritems(self._jobstores):
+            if jobstore_alias in (None, alias):
+                job = store.lookup_job(job_id)
+                if job is not None:
+                    return job, alias
+
+        raise JobLookupError(job_id)
+
+    def _dispatch_event(self, event):
+        """
+        Dispatches the given event to interested listeners.
+
+        :param SchedulerEvent event: the event to send
+        """
+
+        with self._listeners_lock:
+            listeners = tuple(self._listeners)
+
+        for cb, mask in listeners:
+            if event.code & mask:
+                try:
+                    cb(event)
+                except:
+                    self._logger.exception('Error notifying listener')
+
+    def _real_add_job(self, job, jobstore_alias, replace_existing, wakeup):
+        """
+        :param Job job: the job to add
+        :param bool replace_existing: ``True`` to use update_job() in case the job already exists in the store
+        :param bool wakeup: ``True`` to wake up the scheduler after adding the job
+        """
+
+        # Fill in undefined values with defaults
+        replacements = {}
+        for key, value in six.iteritems(self._job_defaults):
+            if not hasattr(job, key):
+                replacements[key] = value
+
+        # Calculate the next run time if there is none defined
+        if not hasattr(job, 'next_run_time'):
+            now = datetime.now(self.timezone)
+            replacements['next_run_time'] = job.trigger.get_next_fire_time(None, now)
+
+        # Apply any replacements
+        job._modify(**replacements)
+
+        # Add the job to the given job store
+        store = self._lookup_jobstore(jobstore_alias)
+        try:
+            store.add_job(job)
+        except ConflictingIdError:
+            if replace_existing:
+                store.update_job(job)
+            else:
+                raise
+
+        # Mark the job as no longer pending
+        job._jobstore_alias = jobstore_alias
+
+        # Notify listeners that a new job has been added
+        event = JobEvent(EVENT_JOB_ADDED, job.id, jobstore_alias)
+        self._dispatch_event(event)
+
+        self._logger.info('Added job "%s" to job store "%s"', job.name, jobstore_alias)
+
+        # Notify the scheduler about the new job
+        if wakeup:
+            self.wakeup()
+
+    def _create_plugin_instance(self, type_, alias, constructor_kwargs):
+        """Creates an instance of the given plugin type, loading the plugin first if necessary."""
+
+        plugin_container, class_container, base_class = {
+            'trigger': (self._trigger_plugins, self._trigger_classes, BaseTrigger),
+            'jobstore': (self._jobstore_plugins, self._jobstore_classes, BaseJobStore),
+            'executor': (self._executor_plugins, self._executor_classes, BaseExecutor)
+        }[type_]
+
+        try:
+            plugin_cls = class_container[alias]
+        except KeyError:
+            if alias in plugin_container:
+                plugin_cls = class_container[alias] = plugin_container[alias].load()
+                if not issubclass(plugin_cls, base_class):
+                    raise TypeError('The {0} entry point does not point to a {0} class'.format(type_))
+            else:
+                raise LookupError('No {0} by the name "{1}" was found'.format(type_, alias))
+
+        return plugin_cls(**constructor_kwargs)
+
+    def _create_trigger(self, trigger, trigger_args):
+        if isinstance(trigger, BaseTrigger):
+            return trigger
+        elif trigger is None:
+            trigger = 'date'
+        elif not isinstance(trigger, six.string_types):
+            raise TypeError('Expected a trigger instance or string, got %s instead' % trigger.__class__.__name__)
+
+        # Use the scheduler's time zone if nothing else is specified
+        trigger_args.setdefault('timezone', self.timezone)
+
+        # Instantiate the trigger class
+        return self._create_plugin_instance('trigger', trigger, trigger_args)
+
+    def _create_lock(self):
+        """Creates a reentrant lock object."""
+
+        return RLock()
+
+    def _process_jobs(self):
+        """
+        Iterates through jobs in every jobstore, starts jobs that are due and figures out how long to wait for the next
+        round.
+        """
+
+        self._logger.debug('Looking for jobs to run')
+        now = datetime.now(self.timezone)
+        next_wakeup_time = None
+
+        with self._jobstores_lock:
+            for jobstore_alias, jobstore in six.iteritems(self._jobstores):
+                for job in jobstore.get_due_jobs(now):
+                    # Look up the job's executor
+                    try:
+                        executor = self._lookup_executor(job.executor)
+                    except:
+                        self._logger.error(
+                            'Executor lookup ("%s") failed for job "%s" -- removing it from the job store',
+                            job.executor, job)
+                        self.remove_job(job.id, jobstore_alias)
+                        continue
+
+                    run_times = job._get_run_times(now)
+                    run_times = run_times[-1:] if run_times and job.coalesce else run_times
+                    if run_times:
+                        try:
+                            executor.submit_job(job, run_times)
+                        except MaxInstancesReachedError:
+                            self._logger.warning(
+                                'Execution of job "%s" skipped: maximum number of running instances reached (%d)',
+                                job, job.max_instances)
+                        except:
+                            self._logger.exception('Error submitting job "%s" to executor "%s"', job, job.executor)
+
+                        # Update the job if it has a next execution time. Otherwise remove it from the job store.
+                        job_next_run = job.trigger.get_next_fire_time(run_times[-1], now)
+                        if job_next_run:
+                            job._modify(next_run_time=job_next_run)
+                            jobstore.update_job(job)
+                        else:
+                            self.remove_job(job.id, jobstore_alias)
+
+                # Set a new next wakeup time if there isn't one yet or the jobstore has an even earlier one
+                jobstore_next_run_time = jobstore.get_next_run_time()
+                if jobstore_next_run_time and (next_wakeup_time is None or jobstore_next_run_time < next_wakeup_time):
+                    next_wakeup_time = jobstore_next_run_time
+
+        # Determine the delay until this method should be called again
+        if next_wakeup_time is not None:
+            wait_seconds = max(timedelta_seconds(next_wakeup_time - now), 0)
+            self._logger.debug('Next wakeup is due at %s (in %f seconds)', next_wakeup_time, wait_seconds)
+        else:
+            wait_seconds = None
+            self._logger.debug('No jobs; waiting until a job is added')
+
+        return wait_seconds

lib/apscheduler/schedulers/blocking.py

@@ -0,0 +1,32 @@
+from __future__ import absolute_import
+from threading import Event
+
+from apscheduler.schedulers.base import BaseScheduler
+
+
+class BlockingScheduler(BaseScheduler):
+    """
+    A scheduler that runs in the foreground (:meth:`~apscheduler.schedulers.base.BaseScheduler.start` will block).
+    """
+
+    MAX_WAIT_TIME = 4294967  # Maximum value accepted by Event.wait() on Windows
+
+    _event = None
+
+    def start(self):
+        super(BlockingScheduler, self).start()
+        self._event = Event()
+        self._main_loop()
+
+    def shutdown(self, wait=True):
+        super(BlockingScheduler, self).shutdown(wait)
+        self._event.set()
+
+    def _main_loop(self):
+        while self.running:
+            wait_seconds = self._process_jobs()
+            self._event.wait(wait_seconds if wait_seconds is not None else self.MAX_WAIT_TIME)
+            self._event.clear()
+
+    def wakeup(self):
+        self._event.set()

lib/apscheduler/schedulers/gevent.py

@@ -0,0 +1,35 @@
+from __future__ import absolute_import
+
+from apscheduler.schedulers.blocking import BlockingScheduler
+from apscheduler.schedulers.base import BaseScheduler
+
+try:
+    from gevent.event import Event
+    from gevent.lock import RLock
+    import gevent
+except ImportError:  # pragma: nocover
+    raise ImportError('GeventScheduler requires gevent installed')
+
+
+class GeventScheduler(BlockingScheduler):
+    """A scheduler that runs as a Gevent greenlet."""
+
+    _greenlet = None
+
+    def start(self):
+        BaseScheduler.start(self)
+        self._event = Event()
+        self._greenlet = gevent.spawn(self._main_loop)
+        return self._greenlet
+
+    def shutdown(self, wait=True):
+        super(GeventScheduler, self).shutdown(wait)
+        self._greenlet.join()
+        del self._greenlet
+
+    def _create_lock(self):
+        return RLock()
+
+    def _create_default_executor(self):
+        from apscheduler.executors.gevent import GeventExecutor
+        return GeventExecutor()

lib/apscheduler/schedulers/qt.py

@@ -0,0 +1,46 @@
+from __future__ import absolute_import
+
+from apscheduler.schedulers.base import BaseScheduler
+
+try:
+    from PyQt5.QtCore import QObject, QTimer
+except ImportError:  # pragma: nocover
+    try:
+        from PyQt4.QtCore import QObject, QTimer
+    except ImportError:
+        try:
+            from PySide.QtCore import QObject, QTimer  # flake8: noqa
+        except ImportError:
+            raise ImportError('QtScheduler requires either PyQt5, PyQt4 or PySide installed')
+
+
+class QtScheduler(BaseScheduler):
+    """A scheduler that runs in a Qt event loop."""
+
+    _timer = None
+
+    def start(self):
+        super(QtScheduler, self).start()
+        self.wakeup()
+
+    def shutdown(self, wait=True):
+        super(QtScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._timer = QTimer.singleShot(wait_seconds * 1000, self._process_jobs)
+
+    def _stop_timer(self):
+        if self._timer:
+            if self._timer.isActive():
+                self._timer.stop()
+            del self._timer
+
+    def wakeup(self):
+        self._start_timer(0)
+
+    def _process_jobs(self):
+        wait_seconds = super(QtScheduler, self)._process_jobs()
+        self._start_timer(wait_seconds)

lib/apscheduler/schedulers/tornado.py

@@ -0,0 +1,60 @@
+from __future__ import absolute_import
+from datetime import timedelta
+from functools import wraps
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.util import maybe_ref
+
+try:
+    from tornado.ioloop import IOLoop
+except ImportError:  # pragma: nocover
+    raise ImportError('TornadoScheduler requires tornado installed')
+
+
+def run_in_ioloop(func):
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        self._ioloop.add_callback(func, self, *args, **kwargs)
+    return wrapper
+
+
+class TornadoScheduler(BaseScheduler):
+    """
+    A scheduler that runs on a Tornado IOLoop.
+
+    =========== ===============================================================
+    ``io_loop`` Tornado IOLoop instance to use (defaults to the global IO loop)
+    =========== ===============================================================
+    """
+
+    _ioloop = None
+    _timeout = None
+
+    def start(self):
+        super(TornadoScheduler, self).start()
+        self.wakeup()
+
+    @run_in_ioloop
+    def shutdown(self, wait=True):
+        super(TornadoScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _configure(self, config):
+        self._ioloop = maybe_ref(config.pop('io_loop', None)) or IOLoop.current()
+        super(TornadoScheduler, self)._configure(config)
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._timeout = self._ioloop.add_timeout(timedelta(seconds=wait_seconds), self.wakeup)
+
+    def _stop_timer(self):
+        if self._timeout:
+            self._ioloop.remove_timeout(self._timeout)
+            del self._timeout
+
+    @run_in_ioloop
+    def wakeup(self):
+        self._stop_timer()
+        wait_seconds = self._process_jobs()
+        self._start_timer(wait_seconds)

lib/apscheduler/schedulers/twisted.py

@@ -0,0 +1,65 @@
+from __future__ import absolute_import
+from functools import wraps
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.util import maybe_ref
+
+try:
+    from twisted.internet import reactor as default_reactor
+except ImportError:  # pragma: nocover
+    raise ImportError('TwistedScheduler requires Twisted installed')
+
+
+def run_in_reactor(func):
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        self._reactor.callFromThread(func, self, *args, **kwargs)
+    return wrapper
+
+
+class TwistedScheduler(BaseScheduler):
+    """
+    A scheduler that runs on a Twisted reactor.
+
+    Extra options:
+
+    =========== ========================================================
+    ``reactor`` Reactor instance to use (defaults to the global reactor)
+    =========== ========================================================
+    """
+
+    _reactor = None
+    _delayedcall = None
+
+    def _configure(self, config):
+        self._reactor = maybe_ref(config.pop('reactor', default_reactor))
+        super(TwistedScheduler, self)._configure(config)
+
+    def start(self):
+        super(TwistedScheduler, self).start()
+        self.wakeup()
+
+    @run_in_reactor
+    def shutdown(self, wait=True):
+        super(TwistedScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._delayedcall = self._reactor.callLater(wait_seconds, self.wakeup)
+
+    def _stop_timer(self):
+        if self._delayedcall and self._delayedcall.active():
+            self._delayedcall.cancel()
+            del self._delayedcall
+
+    @run_in_reactor
+    def wakeup(self):
+        self._stop_timer()
+        wait_seconds = self._process_jobs()
+        self._start_timer(wait_seconds)
+
+    def _create_default_executor(self):
+        from apscheduler.executors.twisted import TwistedExecutor
+        return TwistedExecutor()

lib/apscheduler/threadpool.py

@@ -1,133 +0,0 @@
-"""
-Generic thread pool class. Modeled after Java's ThreadPoolExecutor.
-Please note that this ThreadPool does *not* fully implement the PEP 3148
-ThreadPool!
-"""
-
-from threading import Thread, Lock, currentThread
-from weakref import ref
-import logging
-import atexit
-
-try:
-    from queue import Queue, Empty
-except ImportError:
-    from Queue import Queue, Empty
-
-logger = logging.getLogger(__name__)
-_threadpools = set()
-
-
-# Worker threads are daemonic in order to let the interpreter exit without
-# an explicit shutdown of the thread pool. The following trick is necessary
-# to allow worker threads to finish cleanly.
-def _shutdown_all():
-    for pool_ref in tuple(_threadpools):
-        pool = pool_ref()
-        if pool:
-            pool.shutdown()
-
-atexit.register(_shutdown_all)
-
-
-class ThreadPool(object):
-    def __init__(self, core_threads=0, max_threads=20, keepalive=1):
-        """
-        :param core_threads: maximum number of persistent threads in the pool
-        :param max_threads: maximum number of total threads in the pool
-        :param thread_class: callable that creates a Thread object
-        :param keepalive: seconds to keep non-core worker threads waiting
-            for new tasks
-        """
-        self.core_threads = core_threads
-        self.max_threads = max(max_threads, core_threads, 1)
-        self.keepalive = keepalive
-        self._queue = Queue()
-        self._threads_lock = Lock()
-        self._threads = set()
-        self._shutdown = False
-
-        _threadpools.add(ref(self))
-        logger.info('Started thread pool with %d core threads and %s maximum '
-                    'threads', core_threads, max_threads or 'unlimited')
-
-    def _adjust_threadcount(self):
-        self._threads_lock.acquire()
-        try:
-            if self.num_threads < self.max_threads:
-                self._add_thread(self.num_threads < self.core_threads)
-        finally:
-            self._threads_lock.release()
-
-    def _add_thread(self, core):
-        t = Thread(target=self._run_jobs, args=(core,))
-        t.setDaemon(True)
-        t.start()
-        self._threads.add(t)
-
-    def _run_jobs(self, core):
-        logger.debug('Started worker thread')
-        block = True
-        timeout = None
-        if not core:
-            block = self.keepalive > 0
-            timeout = self.keepalive
-
-        while True:
-            try:
-                func, args, kwargs = self._queue.get(block, timeout)
-            except Empty:
-                break
-
-            if self._shutdown:
-                break
-
-            try:
-                func(*args, **kwargs)
-            except:
-                logger.exception('Error in worker thread')
-
-        self._threads_lock.acquire()
-        self._threads.remove(currentThread())
-        self._threads_lock.release()
-
-        logger.debug('Exiting worker thread')
-
-    @property
-    def num_threads(self):
-        return len(self._threads)
-
-    def submit(self, func, *args, **kwargs):
-        if self._shutdown:
-            raise RuntimeError('Cannot schedule new tasks after shutdown')
-
-        self._queue.put((func, args, kwargs))
-        self._adjust_threadcount()
-
-    def shutdown(self, wait=True):
-        if self._shutdown:
-            return
-
-        logging.info('Shutting down thread pool')
-        self._shutdown = True
-        _threadpools.remove(ref(self))
-
-        self._threads_lock.acquire()
-        for _ in range(self.num_threads):
-            self._queue.put((None, None, None))
-        self._threads_lock.release()
-
-        if wait:
-            self._threads_lock.acquire()
-            threads = tuple(self._threads)
-            self._threads_lock.release()
-            for thread in threads:
-                thread.join()
-
-    def __repr__(self):
-        if self.max_threads:
-            threadcount = '%d/%d' % (self.num_threads, self.max_threads)
-        else:
-            threadcount = '%d' % self.num_threads
-
-        return '<ThreadPool at %x; threads=%s>' % (id(self), threadcount)

lib/apscheduler/triggers/__init__.py

@@ -1,3 +0,0 @@
-from apscheduler.triggers.cron import CronTrigger
-from apscheduler.triggers.interval import IntervalTrigger
-from apscheduler.triggers.simple import SimpleTrigger

lib/apscheduler/triggers/base.py

@@ -0,0 +1,16 @@
+from abc import ABCMeta, abstractmethod
+
+import six
+
+
+class BaseTrigger(six.with_metaclass(ABCMeta)):
+    """Abstract base class that defines the interface that every trigger must implement."""
+
+    @abstractmethod
+    def get_next_fire_time(self, previous_fire_time, now):
+        """
+        Returns the next datetime to fire on, If no such datetime can be calculated, returns ``None``.
+
+        :param datetime.datetime previous_fire_time: the previous time the trigger was fired
+        :param datetime.datetime now: current datetime
+        """

lib/apscheduler/triggers/cron/__init__.py

@@ -1,32 +1,71 @@
-from datetime import date, datetime
+from datetime import datetime, timedelta
 
-from apscheduler.triggers.cron.fields import *
+from tzlocal import get_localzone
-from apscheduler.util import datetime_ceil, convert_to_datetime
+import six
+
+from apscheduler.triggers.base import BaseTrigger
+from apscheduler.triggers.cron.fields import BaseField, WeekField, DayOfMonthField, DayOfWeekField, DEFAULT_VALUES
+from apscheduler.util import datetime_ceil, convert_to_datetime, datetime_repr, astimezone
 
 
-class CronTrigger(object):
+class CronTrigger(BaseTrigger):
-    FIELD_NAMES = ('year', 'month', 'day', 'week', 'day_of_week', 'hour',
+    """
-                   'minute', 'second')
+    Triggers when current time matches all specified time constraints, similarly to how the UNIX cron scheduler works.
-    FIELDS_MAP = {'year': BaseField,
-                  'month': BaseField,
-                  'week': WeekField,
-                  'day': DayOfMonthField,
-                  'day_of_week': DayOfWeekField,
-                  'hour': BaseField,
-                  'minute': BaseField,
-                  'second': BaseField}
 
-    def __init__(self, **values):
+    :param int|str year: 4-digit year
-        self.start_date = values.pop('start_date', None)
+    :param int|str month: month (1-12)
-        if self.start_date:
+    :param int|str day: day of the (1-31)
-            self.start_date = convert_to_datetime(self.start_date)
+    :param int|str week: ISO week (1-53)
+    :param int|str day_of_week: number or name of weekday (0-6 or mon,tue,wed,thu,fri,sat,sun)
+    :param int|str hour: hour (0-23)
+    :param int|str minute: minute (0-59)
+    :param int|str second: second (0-59)
+    :param datetime|str start_date: earliest possible date/time to trigger on (inclusive)
+    :param datetime|str end_date: latest possible date/time to trigger on (inclusive)
+    :param datetime.tzinfo|str timezone: time zone to use for the date/time calculations
+                                         (defaults to scheduler timezone)
 
+    .. note:: The first weekday is always **monday**.
+    """
+
+    FIELD_NAMES = ('year', 'month', 'day', 'week', 'day_of_week', 'hour', 'minute', 'second')
+    FIELDS_MAP = {
+        'year': BaseField,
+        'month': BaseField,
+        'week': WeekField,
+        'day': DayOfMonthField,
+        'day_of_week': DayOfWeekField,
+        'hour': BaseField,
+        'minute': BaseField,
+        'second': BaseField
+    }
+
+    __slots__ = 'timezone', 'start_date', 'end_date', 'fields'
+
+    def __init__(self, year=None, month=None, day=None, week=None, day_of_week=None, hour=None, minute=None,
+                 second=None, start_date=None, end_date=None, timezone=None):
+        if timezone:
+            self.timezone = astimezone(timezone)
+        elif start_date and start_date.tzinfo:
+            self.timezone = start_date.tzinfo
+        elif end_date and end_date.tzinfo:
+            self.timezone = end_date.tzinfo
+        else:
+            self.timezone = get_localzone()
+
+        self.start_date = convert_to_datetime(start_date, self.timezone, 'start_date')
+        self.end_date = convert_to_datetime(end_date, self.timezone, 'end_date')
+
+        values = dict((key, value) for (key, value) in six.iteritems(locals())
+                      if key in self.FIELD_NAMES and value is not None)
         self.fields = []
+        assign_defaults = False
         for field_name in self.FIELD_NAMES:
             if field_name in values:
                 exprs = values.pop(field_name)
                 is_default = False
-            elif not values:
+                assign_defaults = not values
+            elif assign_defaults:
                 exprs = DEFAULT_VALUES[field_name]
                 is_default = True
             else:
@@ -39,18 +78,16 @@ class CronTrigger(object):
 
     def _increment_field_value(self, dateval, fieldnum):
         """
-        Increments the designated field and resets all less significant fields
+        Increments the designated field and resets all less significant fields to their minimum values.
-        to their minimum values.
 
         :type dateval: datetime
         :type fieldnum: int
-        :type amount: int
+        :return: a tuple containing the new date, and the number of the field that was actually incremented
         :rtype: tuple
-        :return: a tuple containing the new date, and the number of the field
-                 that was actually incremented
         """
-        i = 0
+
         values = {}
+        i = 0
         while i < len(self.fields):
             field = self.fields[i]
             if not field.REAL:
@@ -77,7 +114,8 @@ class CronTrigger(object):
                     values[field.name] = value + 1
                     i += 1
 
-        return datetime(**values), fieldnum
+        difference = datetime(**values) - dateval.replace(tzinfo=None)
+        return self.timezone.normalize(dateval + difference), fieldnum
 
     def _set_field_value(self, dateval, fieldnum, new_value):
         values = {}
@@ -90,13 +128,17 @@ class CronTrigger(object):
                 else:
                     values[field.name] = new_value
 
-        return datetime(**values)
+        difference = datetime(**values) - dateval.replace(tzinfo=None)
+        return self.timezone.normalize(dateval + difference)
+
+    def get_next_fire_time(self, previous_fire_time, now):
+        if previous_fire_time:
+            start_date = max(now, previous_fire_time + timedelta(microseconds=1))
+        else:
+            start_date = max(now, self.start_date) if self.start_date else now
 
-    def get_next_fire_time(self, start_date):
-        if self.start_date:
-            start_date = max(start_date, self.start_date)
-        next_date = datetime_ceil(start_date)
         fieldnum = 0
+        next_date = datetime_ceil(start_date).astimezone(self.timezone)
         while 0 <= fieldnum < len(self.fields):
             field = self.fields[fieldnum]
             curr_value = field.get_value(next_date)
@@ -104,32 +146,31 @@ class CronTrigger(object):
 
             if next_value is None:
                 # No valid value was found
-                next_date, fieldnum = self._increment_field_value(next_date,
+                next_date, fieldnum = self._increment_field_value(next_date, fieldnum - 1)
-                                                                  fieldnum - 1)
             elif next_value > curr_value:
                 # A valid, but higher than the starting value, was found
                 if field.REAL:
-                    next_date = self._set_field_value(next_date, fieldnum,
+                    next_date = self._set_field_value(next_date, fieldnum, next_value)
-                                                      next_value)
                     fieldnum += 1
                 else:
-                    next_date, fieldnum = self._increment_field_value(next_date,
+                    next_date, fieldnum = self._increment_field_value(next_date, fieldnum)
-                                                                      fieldnum)
             else:
                 # A valid value was found, no changes necessary
                 fieldnum += 1
 
+            # Return if the date has rolled past the end date
+            if self.end_date and next_date > self.end_date:
+                return None
+
         if fieldnum >= 0:
             return next_date
 
     def __str__(self):
-        options = ["%s='%s'" % (f.name, str(f)) for f in self.fields
+        options = ["%s='%s'" % (f.name, f) for f in self.fields if not f.is_default]
-                   if not f.is_default]
         return 'cron[%s]' % (', '.join(options))
 
     def __repr__(self):
-        options = ["%s='%s'" % (f.name, str(f)) for f in self.fields
+        options = ["%s='%s'" % (f.name, f) for f in self.fields if not f.is_default]
-                   if not f.is_default]
         if self.start_date:
-            options.append("start_date='%s'" % self.start_date.isoformat(' '))
+            options.append("start_date='%s'" % datetime_repr(self.start_date))
         return '<%s (%s)>' % (self.__class__.__name__, ', '.join(options))

lib/apscheduler/triggers/cron/expressions.py

@@ -7,8 +7,8 @@ import re
 
 from apscheduler.util import asint
 
-__all__ = ('AllExpression', 'RangeExpression', 'WeekdayRangeExpression',
+__all__ = ('AllExpression', 'RangeExpression', 'WeekdayRangeExpression', 'WeekdayPositionExpression',
-           'WeekdayPositionExpression')
+           'LastDayOfMonthExpression')
 
 
 WEEKDAYS = ['mon', 'tue', 'wed', 'thu', 'fri', 'sat', 'sun']
@@ -57,8 +57,7 @@ class RangeExpression(AllExpression):
         if last is None and step is None:
             last = first
         if last is not None and first > last:
-            raise ValueError('The minimum value in a range must not be '
+            raise ValueError('The minimum value in a range must not be higher than the maximum')
-                             'higher than the maximum')
         self.first = first
         self.last = last
 
@@ -102,8 +101,7 @@ class RangeExpression(AllExpression):
 
 
 class WeekdayRangeExpression(RangeExpression):
-    value_re = re.compile(r'(?P<first>[a-z]+)(?:-(?P<last>[a-z]+))?',
+    value_re = re.compile(r'(?P<first>[a-z]+)(?:-(?P<last>[a-z]+))?', re.IGNORECASE)
-                          re.IGNORECASE)
 
     def __init__(self, first, last=None):
         try:
@@ -135,8 +133,7 @@ class WeekdayRangeExpression(RangeExpression):
 
 class WeekdayPositionExpression(AllExpression):
     options = ['1st', '2nd', '3rd', '4th', '5th', 'last']
-    value_re = re.compile(r'(?P<option_name>%s) +(?P<weekday_name>(?:\d+|\w+))'
+    value_re = re.compile(r'(?P<option_name>%s) +(?P<weekday_name>(?:\d+|\w+))' % '|'.join(options), re.IGNORECASE)
-                          % '|'.join(options), re.IGNORECASE)
 
     def __init__(self, option_name, weekday_name):
         try:
@@ -169,10 +166,23 @@ class WeekdayPositionExpression(AllExpression):
             return target_day
 
     def __str__(self):
-        return '%s %s' % (self.options[self.option_num],
+        return '%s %s' % (self.options[self.option_num], WEEKDAYS[self.weekday])
-                          WEEKDAYS[self.weekday])
 
     def __repr__(self):
-        return "%s('%s', '%s')" % (self.__class__.__name__,
+        return "%s('%s', '%s')" % (self.__class__.__name__, self.options[self.option_num], WEEKDAYS[self.weekday])
-                                   self.options[self.option_num],
+
-                                   WEEKDAYS[self.weekday])
+
+class LastDayOfMonthExpression(AllExpression):
+    value_re = re.compile(r'last', re.IGNORECASE)
+
+    def __init__(self):
+        pass
+
+    def get_next_value(self, date, field):
+        return monthrange(date.year, date.month)[1]
+
+    def __str__(self):
+        return 'last'
+
+    def __repr__(self):
+        return "%s()" % self.__class__.__name__

lib/apscheduler/triggers/cron/fields.py

@@ -5,18 +5,18 @@ fields.
 
 from calendar import monthrange
 
-from apscheduler.triggers.cron.expressions import *
+from apscheduler.triggers.cron.expressions import (
-
+    AllExpression, RangeExpression, WeekdayPositionExpression, LastDayOfMonthExpression, WeekdayRangeExpression)
-__all__ = ('MIN_VALUES', 'MAX_VALUES', 'DEFAULT_VALUES', 'BaseField',
-           'WeekField', 'DayOfMonthField', 'DayOfWeekField')
 
 
-MIN_VALUES = {'year': 1970, 'month': 1, 'day': 1, 'week': 1,
+__all__ = ('MIN_VALUES', 'MAX_VALUES', 'DEFAULT_VALUES', 'BaseField', 'WeekField', 'DayOfMonthField', 'DayOfWeekField')
-              'day_of_week': 0, 'hour': 0, 'minute': 0, 'second': 0}
+
-MAX_VALUES = {'year': 2 ** 63, 'month': 12, 'day:': 31, 'week': 53,
+
-              'day_of_week': 6, 'hour': 23, 'minute': 59, 'second': 59}
+MIN_VALUES = {'year': 1970, 'month': 1, 'day': 1, 'week': 1, 'day_of_week': 0, 'hour': 0, 'minute': 0, 'second': 0}
-DEFAULT_VALUES = {'year': '*', 'month': 1, 'day': 1, 'week': '*',
+MAX_VALUES = {'year': 2 ** 63, 'month': 12, 'day:': 31, 'week': 53, 'day_of_week': 6, 'hour': 23, 'minute': 59,
-                  'day_of_week': '*', 'hour': 0, 'minute': 0, 'second': 0}
+              'second': 59}
+DEFAULT_VALUES = {'year': '*', 'month': 1, 'day': 1, 'week': '*', 'day_of_week': '*', 'hour': 0, 'minute': 0,
+                  'second': 0}
 
 
 class BaseField(object):
@@ -65,16 +65,14 @@ class BaseField(object):
                 self.expressions.append(compiled_expr)
                 return
 
-        raise ValueError('Unrecognized expression "%s" for field "%s"' %
+        raise ValueError('Unrecognized expression "%s" for field "%s"' % (expr, self.name))
-                         (expr, self.name))
 
     def __str__(self):
         expr_strings = (str(e) for e in self.expressions)
         return ','.join(expr_strings)
 
     def __repr__(self):
-        return "%s('%s', '%s')" % (self.__class__.__name__, self.name,
+        return "%s('%s', '%s')" % (self.__class__.__name__, self.name, self)
-                                   str(self))
 
 
 class WeekField(BaseField):
@@ -85,7 +83,7 @@ class WeekField(BaseField):
 
 
 class DayOfMonthField(BaseField):
-    COMPILERS = BaseField.COMPILERS + [WeekdayPositionExpression]
+    COMPILERS = BaseField.COMPILERS + [WeekdayPositionExpression, LastDayOfMonthExpression]
 
     def get_max(self, dateval):
         return monthrange(dateval.year, dateval.month)[1]