v0.5.10

This reverts commit 74e071b1d7.

.gitignore

@@ -6,12 +6,13 @@
 *.pyproj
 *.sln
 
-# Logs and databases #
+# Headphones files #
 ######################
 *.log
 *.db*
 *.db-journal
 *.ini
+version.lock
 logs/*
 cache/*
 
@@ -19,6 +20,7 @@ cache/*
 ##################
 *.crt
 *.key
+*.csr
 
 # OS generated files #
 ######################

.pep8

@@ -0,0 +1,17 @@
+[pep8]
+# E111 indentation is not a multiple of four
+# E121 continuation line under-indented for hanging indent
+# E122 continuation line missing indentation or outdented
+# E124 closing bracket does not match visual indentation
+# E125 continuation line with same indent as next logical line
+# E126 continuation line over-indented for hanging indent
+# E127 continuation line over-indented for visual indent
+# E128 continuation line under-indented for visual indent
+# E261 at least two spaces before inline comment
+# E262 inline comment should start with '# '
+# E265 block comment should start with '# '
+# E302 expected 2 blank lines, found 1
+# E501 line too long (312 > 160 characters)
+# E502 the backslash is redundant between brackets
+ignore = E111,E121,E122,E123,E124,E125,E126,E127,E128,E261,E262,E265,E302,E501,E502
+max-line-length = 160

.travis.yml

@@ -0,0 +1,20 @@
+# Travis CI configuration file
+# http://about.travis-ci.org/docs/
+
+language: python
+
+# Available Python versions:
+# http://about.travis-ci.org/docs/user/ci-environment/#Python-VM-images
+python:
+  - "2.6"
+  - "2.7"
+# pylint 1.4 does not run under python 2.6
+install:
+ - pip install pyOpenSSL
+ - pip install pylint==1.3.1
+ - pip install pyflakes
+ - pip install pep8
+script:
+  - pep8 headphones
+  - pyflakes headphones
+  - nosetests headphones

API.md

@@ -0,0 +1,111 @@
+# API Reference
+The API is still pretty new and needs some serious cleaning up on the backend but should be reasonably functional. There are no error codes yet.
+
+## General structure
+The API endpoint is `http://ip:port + HTTP_ROOT + /api?apikey=$apikey&cmd=$command`
+
+Data response in JSON formatted. If executing a command like "delArtist" or "addArtist" you'll get back an "OK", else, you'll get the data you requested.
+
+## API methods
+
+### getIndex
+Fetch data from index page. Returns: ArtistName, ArtistSortName, ArtistID, Status, DateAdded, [LatestAlbum, ReleaseDate, AlbumID], HaveTracks, TotalTracks, IncludeExtras, LastUpdated, [ArtworkURL, ThumbURL]: a remote url to the artwork/thumbnail.
+
+To get the cached image path, see getArtistArt command. ThumbURL is added/updated when an artist is added/updated. If your using the database method to get the artwork, it's more reliable to use the ThumbURL than the ArtworkURL)
+
+### getArtist&id=$artistid
+Fetch artist data. returns the artist object (see above) and album info: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type, ArtworkURL: hosted image path. For cached image, see getAlbumArt command)
+
+### getAlbum&id=$albumid
+Fetch data from album page. Returns the album object, a description object and a tracks object. Tracks contain: AlbumASIN, AlbumTitle, TrackID, Format, TrackDuration (ms), ArtistName, TrackTitle, AlbumID, ArtistID, Location, TrackNumber, CleanName (stripped of punctuation /styling), BitRate)
+
+### getUpcoming
+Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type
+
+### getWanted
+Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type
+
+### getSimilar
+Returns similar artists  - with a higher "Count" being more likely to be similar. Returns: Count, ArtistName, ArtistID
+
+### getHistory
+Returns: Status, DateAdded, Title, URL (nzb), FolderName, AlbumID, Size (bytes)
+
+### getLogs
+Not working yet
+
+### findArtist&name=$artistname[&limit=$limit]
+Perform artist query on musicbrainz. Returns: url, score, name, uniquename (contains disambiguation info), id)
+
+### findAlbum&name=$albumname[&limit=$limit]
+Perform album query on musicbrainz. Returns: title, url (artist), id (artist), albumurl, albumid, score, uniquename (artist - with disambiguation)
+
+### addArtist&id=$artistid
+Add an artist to the db by artistid)
+
+### addAlbum&id=$releaseid
+Add an album to the db by album release id
+
+### delArtist&id=$artistid
+Delete artist from db by artistid)
+
+### pauseArtist&id=$artistid
+Pause an artist in db)
+
+### resumeArtist&id=$artistid
+Resume an artist in db)
+
+### refreshArtist&id=$artistid
+Refresh info for artist in db from musicbrainz
+
+### queueAlbum&id=$albumid[&new=True&lossless=True]
+Mark an album as wanted and start the searcher. Optional paramters: 'new' looks for new versions, 'lossless' looks only for lossless versions
+
+### unqueueAlbum&id=$albumid
+Unmark album as wanted / i.e. mark as skipped
+
+### forceSearch
+force search for wanted albums - not launched in a separate thread so it may take a bit to complete
+### forceProcess[&dir=/path/to/folder]
+Force post process albums in download directory - also not launched in a separate thread
+### forceActiveArtistsUpdate
+force Active Artist Update - also not launched in a separate thread
+
+### getVersion
+Returns some version information: git_path, install_type, current_version, installed_version, commits_behind
+
+### checkGithub
+Updates the version information above and returns getVersion data
+
+### shutdown
+Shut down headphones
+
+### restart
+Restart headphones
+
+### update
+Update headphones - you may want to check the install type in get version and not allow this if type==exe
+
+### getArtistArt&id=$artistid
+Returns either a relative path to the cached image, or a remote url if the image can't be saved to the cache dir
+
+getAlbumArt&id=$albumid
+see above
+
+### getArtistInfo&id=$artistid
+Returns Summary and Content, both formatted in html.
+
+### getAlbumInfo&id=$albumid
+See above, returns Summary and Content.
+
+### getArtistThumb&id=$artistid
+Returns either a relative path to the cached thumbnail artist image, or an http:// address if the cache dir can't be written to.
+
+### getAlbumThumb&id=$albumid
+See above.
+
+### choose_specific_download&id=$albumid
+Gives you a list of results from searcher.searchforalbum(). Basically runs a normal search, but rather than sorting them and downloading the best result, it dumps the data, which you can then pass on to download_specific_release(). Returns a list of dictionaries with params: title, size, url, provider & kind - all of these values must be passed back to download_specific_release
+
+### download_specific_release&id=albumid&title=$title&size=$size&url=$url&provider=$provider&kind=$kind
+Allows you to manually pass a choose_specific_download release back to searcher.send_to_downloader()

API_REFERENCE

@@ -1,71 +0,0 @@
-The API is still pretty new and needs some serious cleaning up on the backend but should be
-reasonably functional. There are no error codes yet, 
-
-
-General structure:
-http://localhost:8181 + HTTP_ROOT + /api?apikey=$apikey&cmd=$command
-
-Data returned in json format. If executing a command like "delArtist" or "addArtist" you'll get back an "OK", else, you'll get the data you requested
-
-$commands&parameters[&optionalparameters]:
-
-
-getIndex (fetch data from index page. Returns: ArtistName, ArtistSortName, ArtistID, Status, DateAdded,
-												[LatestAlbum, ReleaseDate, AlbumID], HaveTracks, TotalTracks,
-												IncludeExtras, LastUpdated, [ArtworkURL, ThumbURL]: a remote url to the artwork/thumbnail. To get the cached image path, see getArtistArt command.
-                                                                                                    ThumbURL is added/updated when an artist is added/updated. If your using the database method to get the artwork,
-                                                                                                    it's more reliable to use the ThumbURL than the ArtworkURL)
-
-getArtist&id=$artistid (fetch artist data. returns the artist object (see above) and album info: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type, ArtworkURL: hosted image path. For cached image, see getAlbumArt command)
-
-getAlbum&id=$albumid (fetch data from album page. Returns the album object, a description object and a tracks object. Tracks contain: AlbumASIN, AlbumTitle, TrackID, Format, TrackDuration (ms), ArtistName, TrackTitle, AlbumID, ArtistID, Location, TrackNumber, CleanName (stripped of punctuation /styling), BitRate)
-
-getUpcoming (Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type)
-
-getWanted (Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type)
-
-getSimilar (Returns similar artists  - with a higher "Count" being more likely to be similar. Returns: Count, ArtistName, ArtistID)
-
-getHistory (Returns: Status, DateAdded, Title, URL (nzb), FolderName, AlbumID, Size (bytes))
-
-getLogs (not working yet)
-
-findArtist&name=$artistname[&limit=$limit] (perform artist query on musicbrainz. Returns: url, score, name, uniquename (contains disambiguation info), id)
-
-findAlbum&name=$albumname[&limit=$limit]  (perform album query on musicbrainz. Returns: title, url (artist), id (artist), albumurl, albumid, score, uniquename (artist - with disambiguation)
-
-addArtist&id=$artistid (add an artist to the db by artistid)
-addAlbum&id=$releaseid (add an album to the db by album release id)
-
-delArtist&id=$artistid (delete artist from db by artistid)
-
-pauseArtist&id=$artistid (pause an artist in db)
-resumeArtist&id=$artistid (resume an artist in db)
-
-refreshArtist&id=$artistid (refresh info for artist in db from musicbrainz)
-
-queueAlbum&id=$albumid[&new=True&lossless=True] (Mark an album as wanted and start the searcher. Optional paramters: 'new' looks for new versions, 'lossless' looks only for lossless versions)
-unqueueAlbum&id=$albumid (Unmark album as wanted / i.e. mark as skipped)
-
-forceSearch (force search for wanted albums - not launched in a separate thread so it may take a bit to complete)
-forceProcess (force post process albums in download directory - also not launched in a separate thread)
-
-getVersion (Returns some version information: git_path, install_type, current_version, installed_version, commits_behind
-checkGithub (updates the version information above and returns getVersion data)
-
-shutdown (shut down headphones)
-restart (restart headphones)
-update (update headphones - you may want to check the install type in get version and not allow this if type==exe)
-
-getArtistArt&id=$artistid  (Returns either a relative path to the cached image, or a remote url if the image can't be saved to the cache dir)
-getAlbumArt&id=$albumid (see above)
-
-getArtistInfo&id=$artistid (Returns Summary and Content, both formatted in html)
-getAlbumInfo&id=$albumid (See above, returns Summary and Content)
-
-getArtistThumb&id=$artistid (Returns either a relative path to the cached thumbnail artist image, or an http:// address if the cache dir can't be written to)
-getAlbumThumb&id=$albumid (see above)
-
-choose_specific_download&id=$albumid (Gives you a list of results from searcher.searchforalbum(). Basically runs a normal search, but rather than sorting them and downloading the best result, it dumps the data, which you can then pass on to download_specific_release(). Returns a list of dictionaries with params: title, size, url, provider & kind - all of these values must be passed back to download_specific_release)
-
-download_specific_release&id=albumid&title=$title&size=$size&url=$url&provider=$provider&kind=$kind (Allows you to manually pass a choose_specific_download release back to searcher.send_to_downloader())

CHANGELOG.md

@@ -0,0 +1,264 @@
+# Changelog
+
+## v0.5.10
+Released 29 January 2016
+
+Highlights:
+* Added: API option to post-process single folders
+* Added: Ability to specify extension when re-encoding
+* Added: Option to stop renaming folders
+* Fixed: Utorrent torrents not being removed (#2385)
+* Fixed: Torznab to transmission
+* Fixed: Magnet folder names in history
+* Fixed: Multiple torcache fixes
+* Fixed: Updated requests & urllib3 to latest versions to fix errors with pyOpenSSL
+* Improved: Use a temporary folder during post-processing
+* Improved: Added verify_ssl_cert option
+* Improved: Fixed track matching progress
+* Improved: pylint, pep8 & pylint fixes
+* Improved: Stop JS links from scrolling to the top of the page
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.9...v0.5.10).
+
+## v0.5.9
+Released 05 September 2015
+
+Highlights:
+* Added: Providers Strike, Jackett, custom Torznabs
+* Added: Option to stop post-processing if no good match found (#2343)
+* Fixed: Blackhole -> Magnet, limit to torcache
+* Fixed: Kat 403 flac error
+* Fixed: Last.fm errors
+* Fixed: Pushover notifications
+* Improved: Rutracker logging, switched to requests lib
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.8...v0.5.9).
+
+## v0.5.8
+Released 13 July 2015
+
+Highlights:
+* Added: Option to only include official extras
+* Added: Option to wait until album release date before searching
+* Fixed: NotifyMyAndroid notifications
+* Fixed: Plex Notifications
+* Fixed: Metacritic parsing
+* Fixed: Pushbullet notifications
+* Fixed: What.cd not honoring custom search term (#2279)
+* Improved: XSS Search bug
+* Improved: Config page layout
+* Improved: Set localhost as default
+* Improved: Better single artist scanning
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.7...v0.5.8).
+
+## v0.5.7
+Released 01 July 2015
+
+Highlights:
+* Improved: Moved pushover to use requests lib
+* Improved: Plex tokens with Plex Home
+* Improved: Added getLogs & clearLogs to api
+* Improved: Cache MetaCritic scores. Added user-agent header to fix 403 errors
+* Improved: Specify whether to delete folders when force post-processing
+* Improved: Convert target bitrate to vbr preset for what.cd searching
+* Improved: Switched Pushover to requests lib
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.6...v0.5.7).
+
+## v0.5.6
+Released 08 June 2015
+
+Highlights:
+* Added: Metacritic scores
+* Added: Series support (e.g. Cafe Del Mar, Now That's What I Call Music, etc)
+* Added: Filter out clean/edited/censored releases (#2198)
+* Added: Button on the log page to toggle verbose/debug logging
+* Fixed: Connecting to SABnzbd over https with python >= 2.7.9
+* Fixed: Email Notifications with SSL
+* Fixed: Don't limit musicbrainz results to first 100
+* Fixed: nzbget url fix
+* Fixed: OSX Notifications
+* Improved: Cuesplit, allow wav, ape to be split
+* Improved: Moved the 'freeze db' option to the advanced->misc. tab
+* Improved: Moved kickass searching to json api, so it doesn't throw 404 errors anymore when there are no results
+* Improved: SSL for headphones indexer
+* Improved: Disable update dialog box if check_github is diabled
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.5...v0.5.6).
+
+## v0.5.5
+Released 04 May 2015
+
+Highlights:
+* Added: force ID3v2.3 during post processing (#2121)
+* Added: MusicBrainz authentication (#2125)
+* Added: Email notifications (addresses #1045)
+* Fixed: Kickass url updated to kickass.to (#2119)
+* Fixed: Piratebay returning 0 bytes for all files
+* Fixed: Albums stopped automatically refreshing when adding an artist
+* Fixed: Min/max sizes for target bitrate
+* Fixed: Don't filter any results if looking for a specific download
+* Fixed: Sort by size in the specific download table
+* Fixed: Deal with beets recommendation.none correctly
+* Improved: Close dialog window automatically when choosing a specific download
+* Improved: Move some repetitive log messages to debug level
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.4...v0.5.5).
+
+## v0.5.4
+Released 05 February 2015
+
+Highlights:
+* Added: backported 'Scan Artist' feature from `sarakha63/headphones`
+* Fixed: change file permissions of cache files according to settings (#2102)
+* Fixed: hide Songkick if no regional events are available
+* Fixed: only reschedule jobs if timeout changed (#2099)
+* Fixed: limit dialog height (#2096)
+* Improved: upgraded requests to 2.5.1
+* Improved: upgraded unidecode to 0.04.17
+* Improved: upgraded tzlocal to 1.1.2
+* Improved: upgraded PyTZ to 2014.10
+* Improved: upgraded mako to 1.0.1
+* Improved: upgraded mutagen to 1.27
+* Improved: upgraded beets to 1.3.10
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.3...v0.5.4).
+
+## v0.5.3
+Released 15 January 2015
+
+Highlights:
+* Added: update active artists via API (#2075)
+* Fixed: queue instance lacks `add` method (#2055)
+* Improved: better SSL error messages (#2058)
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.2...v0.5.3).
+
+## v0.5.2
+Released 28 December 2014
+
+Highlights:
+* Added: advanced option to ignore certain folders by patterns. (#2037)
+* Added: advanced option to ignore certain files by patterns (library only)
+* Added: specify optional paths to CUE splitting tools (#1938)
+* Added: experimental support for OPB (provide details yourself!)
+* Fixed: magnet to torrent conversion (#1926)
+* Fixed: new KAT URL (#2043)
+* Fixed: LMS notifications (#1564)
+* Improved: notify user of SSL-related warnings, instead of silently failing.
+* Improved: show all search results for 'Choose Specific Release'
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.1...v0.5.2).
+
+## v0.5.1
+Released 13 November 2014
+
+Highlights:
+* Added: allow one to disable interval tasks (#2002)
+* Added: script to downgrade Headphones to last version that started (Linux only)
+* Fixed: SSL issues in CherryPy. Self-generated certificates will be 2048 now (#1995)
+* Fixed: Transmission URL detection (#1998)
+* Fixed: missing dependencies for APScheduler and CherryPy (#2001)
+* Improved: symlink infinite recursion detection
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5...v0.5.1).
+
+## v0.5
+Released 10 November 2014
+
+Highlights:
+* Added: CUE splitter
+* Added: filter search result by MusicBrainz Release Group ID
+* Added: follow symlinks while scanning library (#1953)
+* Fixed: crash during post processing (#1897)
+* Fixed: embedding lyrics (#1896)
+* Fixed: HTTP errors with older versions of Python 2.6
+* Fixed: jump back to top of page (#1948)
+* Improved: parse MusicBrainz RGID first when post processing (#1952)
+* Improved: Growl unicode characters (#1695)
+* Improved: search handling for PB and KAT
+* Improved: Last.FM API support (#1877)
+* Improved: upgraded CherryPy to version 3.6.0
+* Improved: upgraded Requests to version 2.4.1
+* Improved: upgraded APScheduler to version 3.0.1
+* Improved: lot of code refactoring
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.4...v0.5).
+
+## v0.4
+Released 20 September 2014
+
+Highlights:
+* Added: support for libav-tools (which replaces FFmpeg under Ubuntu)
+* Added: option to freeze library when post processing
+* Added: Songkick per area
+* Added: rename original NFO file (#1797)
+* Removed: removed dead search providers
+* Fixed: removed left-overs of old packages
+* Improved: rename by original folder (#1811)
+* Improved: uTorrent fixess
+* Improved: delayed loading of album art
+* Improved: search result parser
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.4...v0.4).
+
+## v0.3.4
+Released 15 May 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.3...v0.3.4).
+
+## v0.3.3
+Released 05 May 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.2...v0.3.3).
+
+## v0.3.2
+Released 17 April 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.1...v0.3.2).
+
+## v0.3.1
+Released 11 April 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3...v0.3.1).
+
+## v0.3
+Released 09 April 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2.3...v0.3).
+
+## v0.2.3
+Released 28 March 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2.2...v0.2.3).
+
+## v0.2.2
+Released 15 January 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2.1...v0.2.2).
+
+## v0.2.1
+Released 13 January 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2...v0.2.1).
+
+## v0.2
+Released 13 January 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.1.2...v0.2).
+
+## v0.1.2
+Released 11 October 2013
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.1.1...v0.1.2).
+
+## v0.1.1
+Released 26 August 2013
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.1...v0.1.1).
+
+## v0.1
+Released 05 August 2013
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/2156e1341405d07c5bcfbe994f6b354b32d94cda...v0.1).

CONTRIBUTING.md

@@ -0,0 +1,52 @@
+# Contributing to Headphones
+
+## Issues
+In case you read this because you are posting an issue, please take a minute and conside the things below. The issue tracker is not a support forum. It is primarily intended to submit bugs, improvements or feature requests. However, we are glad to help you, and make sure the problem is not caused by Headphones, but don't expect step-by-step answers.
+
+* Use the search function. Chances are that your problem is already discussed. Do not append to (closed) issues if your problem does not fit the discussion.
+* Visit the [Troubleshooting](../../wiki/TroubleShooting) wiki first.
+* Use [proper formatting](https://help.github.com/articles/github-flavored-markdown/). Paste your logs in code blocks.
+* Close your issue if you resolved it.
+
+## Pull Requests
+If you think you can contribute code to the Headphones repository, do not hesitate to submit a pull request.
+
+### Branches
+All pull requests should be based on the `develop` branch, to minimize cross merges. When you want to develop a new feature, clone the repository with `git clone origin/develop -b FEATURE_NAME`. Use meaningful commit messages.
+
+### Python Code
+
+#### Compatibility
+The code should work with Python 2.6 and 2.7. Note that Headphones runs on different platforms, including Network Attached Storage devices such as Synology.
+
+Re-use existing code. Do not hesitate to add logging in your code. You can the logger module `headphones.logger.*` for this. Web requests are invoked via `headphones.request.*` and derived ones. Use these methods to automatically add proper and meaningful error handling.
+
+#### Code conventions
+Although Headphones did not adapt a code convention in the past, we try to follow the [PEP8](http://legacy.python.org/dev/peps/pep-0008/) conventions for future code. A short summary to remind you (copied from http://wiki.ros.org/PyStyleGuide):
+
+ * 4 space indentation
+ * 80 characters per line
+ * `package_name`
+ * `ClassName`
+ * `method_name`
+ * `field_name`
+ * `_private_something`
+ * `self.__really_private_field`
+ * `_global`
+
+#### Documentation
+Document your code. Use docstrings See [PEP-257](https://www.python.org/dev/peps/pep-0257/) for more information.
+
+#### Continuous Integration
+Headphones has a configuration file for [travis-ci](https://travis-ci.org/). You can add your forked repo to Travis to have it check your code against PEP8, PyLint, and PyFlakes for you. Your pull request will show a green check mark or a red cross on each tested commit, depending on if linting passes.
+
+### HTML/Template code
+
+#### Compatibility
+HTML5 compatible browsers are targetted. There is no specific mobile version of Headphones yet.
+
+#### Conventions
+* 4 space indentation
+* `methodName`
+* `variableName`
+* `ClassName`

Headphones.py

@@ -14,15 +14,14 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os, sys
+import os
+import sys
 
 # Ensure lib added to path, before any other imports
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib/'))
 
 from headphones import webstart, logger
 
-from configobj import ConfigObj
-
 import locale
 import time
 import signal
@@ -33,6 +32,7 @@ import headphones
 signal.signal(signal.SIGINT, headphones.sig_handler)
 signal.signal(signal.SIGTERM, headphones.sig_handler)
 
+
 def main():
     """
     Headphones application entry point. Parses arguments, setups encoding and
@@ -63,16 +63,24 @@ def main():
         headphones.SYS_ENCODING = 'UTF-8'
 
     # Set up and gather command line arguments
-    parser = argparse.ArgumentParser(description='Music add-on for SABnzbd+, Transmission and more.')
+    parser = argparse.ArgumentParser(
+        description='Music add-on for SABnzbd+, Transmission and more.')
 
-    parser.add_argument('-v', '--verbose', action='store_true', help='Increase console logging verbosity')
-    parser.add_argument('-q', '--quiet', action='store_true', help='Turn off console logging')
-    parser.add_argument('-d', '--daemon', action='store_true', help='Run as a daemon')
-    parser.add_argument('-p', '--port', type=int, help='Force Headphones to run on a specified port')
-    parser.add_argument('--datadir', help='Specify a directory where to store your data files')
+    parser.add_argument(
+        '-v', '--verbose', action='store_true', help='Increase console logging verbosity')
+    parser.add_argument(
+        '-q', '--quiet', action='store_true', help='Turn off console logging')
+    parser.add_argument(
+        '-d', '--daemon', action='store_true', help='Run as a daemon')
+    parser.add_argument(
+        '-p', '--port', type=int, help='Force Headphones to run on a specified port')
+    parser.add_argument(
+        '--datadir', help='Specify a directory where to store your data files')
     parser.add_argument('--config', help='Specify a config file to use')
-    parser.add_argument('--nolaunch', action='store_true', help='Prevent browser from launching on startup')
-    parser.add_argument('--pidfile', help='Create a pid file (only relevant when running as a daemon)')
+    parser.add_argument('--nolaunch', action='store_true',
+                        help='Prevent browser from launching on startup')
+    parser.add_argument(
+        '--pidfile', help='Create a pid file (only relevant when running as a daemon)')
 
     args = parser.parse_args()
 
@@ -81,9 +89,14 @@ def main():
     if args.quiet:
         headphones.QUIET = True
 
+    # Do an intial setup of the logger.
+    logger.initLogger(console=not headphones.QUIET, log_dir=False,
+        verbose=headphones.VERBOSE)
+
     if args.daemon:
         if sys.platform == 'win32':
-            sys.stderr.write("Daemonizing not supported under Windows, starting normally\n")
+            sys.stderr.write(
+                "Daemonizing not supported under Windows, starting normally\n")
         else:
             headphones.DAEMON = True
             headphones.QUIET = True
@@ -91,11 +104,14 @@ def main():
     if args.pidfile:
         headphones.PIDFILE = str(args.pidfile)
 
-        # If the pidfile already exists, headphones may still be running, so exit
+        # If the pidfile already exists, headphones may still be running, so
+        # exit
         if os.path.exists(headphones.PIDFILE):
-            sys.exit("PID file '" + headphones.PIDFILE + "' already exists. Exiting.")
+            raise SystemExit("PID file '%s' already exists. Exiting." %
+                headphones.PIDFILE)
 
-        # The pidfile is only useful in daemon mode, make sure we can write the file properly
+        # The pidfile is only useful in daemon mode, make sure we can write the
+        # file properly
         if headphones.DAEMON:
             headphones.CREATEPID = True
 
@@ -103,9 +119,10 @@ def main():
                 with open(headphones.PIDFILE, 'w') as fp:
                     fp.write("pid\n")
             except IOError as e:
-                raise SystemExit("Unable to write PID file: %s [%d]", e.strerror, e.errno)
+                raise SystemExit("Unable to write PID file: %s", e)
         else:
-            logger.warn("Not running in daemon mode. PID file creation disabled.")
+            logger.warn("Not running in daemon mode. PID file creation " \
+                "disabled.")
 
     # Determine which data directory and config file to use
     if args.datadir:
@@ -114,27 +131,28 @@ def main():
         headphones.DATA_DIR = headphones.PROG_DIR
 
     if args.config:
-        headphones.CONFIG_FILE = args.config
+        config_file = args.config
     else:
-        headphones.CONFIG_FILE = os.path.join(headphones.DATA_DIR, 'config.ini')
+        config_file = os.path.join(headphones.DATA_DIR, 'config.ini')
 
     # Try to create the DATA_DIR if it doesn't exist
     if not os.path.exists(headphones.DATA_DIR):
         try:
             os.makedirs(headphones.DATA_DIR)
         except OSError:
-            raise SystemExit('Could not create data directory: ' + headphones.DATA_DIR + '. Exiting....')
+            raise SystemExit(
+                'Could not create data directory: ' + headphones.DATA_DIR + '. Exiting....')
 
     # Make sure the DATA_DIR is writeable
     if not os.access(headphones.DATA_DIR, os.W_OK):
-        raise SystemExit('Cannot write to the data directory: ' + headphones.DATA_DIR + '. Exiting...')
+        raise SystemExit(
+            'Cannot write to the data directory: ' + headphones.DATA_DIR + '. Exiting...')
 
     # Put the database in the DATA_DIR
     headphones.DB_FILE = os.path.join(headphones.DATA_DIR, 'headphones.db')
-    headphones.CFG = ConfigObj(headphones.CONFIG_FILE, encoding='utf-8')
 
     # Read config and start logging
-    headphones.initialize()
+    headphones.initialize(config_file)
 
     if headphones.DAEMON:
         headphones.daemonize()
@@ -147,28 +165,40 @@ def main():
         http_port = args.port
         logger.info('Using forced web server port: %i', http_port)
     else:
-        http_port = int(headphones.HTTP_PORT)
+        http_port = int(headphones.CONFIG.HTTP_PORT)
+
+    # Check if pyOpenSSL is installed. It is required for certificate generation
+    # and for CherryPy.
+    if headphones.CONFIG.ENABLE_HTTPS:
+        try:
+            import OpenSSL
+        except ImportError:
+            logger.warn("The pyOpenSSL module is missing. Install this " \
+                "module to enable HTTPS. HTTPS will be disabled.")
+            headphones.CONFIG.ENABLE_HTTPS = False
 
     # Try to start the server. Will exit here is address is already in use.
-    webstart.initialize({
+    web_config = {
         'http_port': http_port,
-        'http_host': headphones.HTTP_HOST,
-        'http_root': headphones.HTTP_ROOT,
-        'http_proxy': headphones.HTTP_PROXY,
-        'enable_https': headphones.ENABLE_HTTPS,
-        'https_cert': headphones.HTTPS_CERT,
-        'https_key': headphones.HTTPS_KEY,
-        'http_username': headphones.HTTP_USERNAME,
-        'http_password': headphones.HTTP_PASSWORD,
-    })
-
-    if headphones.LAUNCH_BROWSER and not args.nolaunch:
-        headphones.launch_browser(headphones.HTTP_HOST, http_port,
-            headphones.HTTP_ROOT)
+        'http_host': headphones.CONFIG.HTTP_HOST,
+        'http_root': headphones.CONFIG.HTTP_ROOT,
+        'http_proxy': headphones.CONFIG.HTTP_PROXY,
+        'enable_https': headphones.CONFIG.ENABLE_HTTPS,
+        'https_cert': headphones.CONFIG.HTTPS_CERT,
+        'https_key': headphones.CONFIG.HTTPS_KEY,
+        'http_username': headphones.CONFIG.HTTP_USERNAME,
+        'http_password': headphones.CONFIG.HTTP_PASSWORD,
+    }
+    webstart.initialize(web_config)
 
     # Start the background threads
     headphones.start()
 
+    # Open webbrowser
+    if headphones.CONFIG.LAUNCH_BROWSER and not args.nolaunch:
+        headphones.launch_browser(headphones.CONFIG.HTTP_HOST, http_port,
+                                  headphones.CONFIG.HTTP_ROOT)
+
     # Wait endlessy for a signal to happen
     while True:
         if not headphones.SIGNAL:
@@ -190,4 +220,4 @@ def main():
 
 # Call main()
 if __name__ == "__main__":
-    main()
+    main()

README.md

@@ -1,24 +1,28 @@
-#![preview thumb](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png)Headphones
+##![Headphones Logo](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png) Headphones
 
-###Support & Discuss
+**Master Branch:**  [![Build Status](https://travis-ci.org/rembo10/headphones.svg?branch=master)](https://travis-ci.org/rembo10/headphones)
+**Develop Branch:** [![Build Status](https://travis-ci.org/rembo10/headphones.svg?branch=develop)](https://travis-ci.org/rembo10/headphones)
 
-You are free to join the HP support community on IRC where you can ask questions, hang around and discuss anything related to HP.
+Headphones is an automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, µTorrent and Blackhole.
 
-1. Use any IRC client and connect to the Freenode server.
-2. Join #headphones
+## Support & Discuss
+You are free to join the Headphones support community on IRC where you can ask questions, hang around and discuss anything related to HP.
 
-###Installation and Notes
+1. Use any IRC client and connect to the Freenode server, `irc.freenode.net`.
+2. Join the `#headphones` channel.
 
-[Read our Wiki](../../wiki) on how to install and use HeadPhones properly.
+## Installation and Notes
 
-[**Troubleshooting** page](../../wiki/TroubleShooting) in the wiki can help you with comon problems.
+* [Installation page](../../wiki/Installation) shows you how to install Headphones.
+* [Usage guide](../../wiki/Usage-guide) introduces you to Headphones.
+* [Troubleshooting page](../../wiki/TroubleShooting) in the wiki can help you with common problems.
 
 **Issues** can be reported on the GitHub issue tracker considering these rules:
 
 1. Analyze your log, you just might find the solution yourself!
 2. You read the wiki and searched existing issues, but this is not solving your problem.
-3. Post the issue with a clear title, description and the HP log and use [proper markdown syntax](https://help.github.com/articles/github-flavored-markdown) to structure your text (code/log in code blocks). 
-4. Close your issue when it's solved! If you found the solution yourself please comment so that others benefit from it.
+3. Post the issue with a clear title, description and the HP log and use [proper markdown syntax](https://help.github.com/articles/github-flavored-markdown) to structure your text (code/log in code blocks).
+4. Close your issue when it's solved! If you found the solution yourself, please comment so that others benefit from it.
 
 **Feature requests** can be reported on the GitHub issue tracker too:
 
@@ -27,17 +31,15 @@ You are free to join the HP support community on IRC where you can ask questions
 
 If you **comply with these rules** you can [post your request/issue](http://github.com/rembo10/headphones/issues).
 
-**Support** the project by implementing new features, solving support tickets and provide bug fixes. 
-If you change something in the code always make a PR to the developer branch instead of the master branch.
+**Support** the project by implementing new features, solving support tickets and provide bug fixes.
 
+## Screenshots
 
-###Screenshots
-
-Homepage (Artist Overview)
+Homepage (Artist Overview):
 
 ![preview thumb](http://i.imgur.com/LZO9a.png)
 
-One of the many settings pages....
+One of the many settings pages:
 
 ![preview thumb](http://i.imgur.com/xcWNy.png)
 
@@ -49,7 +51,7 @@ Import Your Favorite Artists:
 
 ![preview thumb](http://i.imgur.com/6tZoC.png)
 
-Artist Search Results (also search by album!):
+Search Results:
 
 ![preview thumb](http://i.imgur.com/rIV0P.png)
 
@@ -61,5 +63,5 @@ Album Page with track overview:
 
 ![preview thumb](http://i.imgur.com/kcjES.png)
 
-
-This is free software under the GPL v3 open source license - so feel free to do with it what you wish.
+## License
+This is free software under the GPL v3 open source license. Feel free to do with it what you wish, but any modification must be open sourced. A copy of the license is included.

contrib/clean_pyc.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# Display information
+echo "This script will remove *.pyc files. These files are generated by Python, but they can cause conflicts after an upgrade. It's safe to remove them, because they will be regenerated."
+echo "Press enter to continue, or CTRL + C to quit."
+read
+
+# Remove the *.pyc
+find "`dirname $0`/.." -type f -name "*.pyc" -exec rm -rf {} \;

contrib/downgrade.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+
+# Parameter check
+if [ -z "$1" ]; then
+    echo "Syntax: $0 <data directory>"
+    exit 1
+fi
+
+# Version file check
+if [ ! -s "$1/version.lock" ]; then
+    echo "Missing the version.lock file in the data folder, or the file is empty. Did you start Headphones at least once?"
+    exit 1
+fi
+
+# Git installation check
+if [ ! -x "$(command -v git)" ]; then
+    echo "Git is required to downgrade."
+    exit 1
+fi
+
+# Display information
+HASH=$(cat $1/version.lock)
+
+echo "This script will try to downgrade Headphones to the last version that started, version $HASH. Make sure you have a backup of your config file and database, just in case!"
+echo "Press enter to continue, or CTRL + C to quit."
+read
+
+# Downgrade
+cd "`dirname $0`/.."
+git reset --hard "$HASH"
+
+echo "All done, Headphones should be downgraded to the last version that started."

contrib/sni_test.py

@@ -0,0 +1,145 @@
+#!/usr/bin/env python
+
+import os
+import sys
+
+# Ensure that we use the Headphones provided libraries.
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "../lib"))
+
+import urlparse
+
+
+def can_import(module):
+    """
+    Return True if a given module can be imported or not.
+    """
+
+    try:
+        __import__(module)
+    except ImportError:
+        return False
+
+    # Module can be imported
+    return True
+
+
+def check_installation():
+    """
+    Check if some core modules are available. Info is based on this topic:
+    https://github.com/rembo10/headphones/issues/2210.
+    """
+
+    if can_import("requests"):
+        import requests
+        requests_version = requests.__version__
+    else:
+        requests_version = "no"
+
+    if can_import("OpenSSL"):
+        import OpenSSL
+        openssl_version = OpenSSL.__version__
+    else:
+        openssl_version = "no"
+
+    if can_import("cryptography"):
+        import cryptography
+        cryptography_version = cryptography.__version__
+    else:
+        cryptography_version = "no"
+
+    if can_import("pyasn1"):
+        import pyasn1
+        pyasn1_version = pyasn1.__version__
+    else:
+        pyasn1_version = "no"
+
+    if can_import("ndg.httpsclient"):
+        from ndg import httpsclient
+        ndg_version = httpsclient.__date__
+    else:
+        ndg_version = "no"
+
+    # Print some system information.
+    sys.stdout.write(
+        "* Checking Python version: %s.%s.%s\n" % sys.version_info[:3])
+    sys.stdout.write("* Operating system: %s\n" % sys.platform)
+
+    sys.stdout.write(
+        "* Checking if requests can be imported: %s\n" % requests_version)
+    sys.stdout.write(
+        "* Checking if pyOpenSSL is installed: %s\n" % openssl_version)
+    sys.stdout.write(
+        "* Checking if cryptography is installed: %s\n" % cryptography_version)
+    sys.stdout.write(
+        "* Checking if pyasn1 is installed: %s\n" % pyasn1_version)
+    sys.stdout.write(
+        "* Checking if ndg.httpsclient is installed: %s\n" % ndg_version)
+
+
+def main():
+    """
+    Test if the current Headphones installation can connect to SNI-enabled
+    servers.
+    """
+
+    # Read the URL to test.
+    if len(sys.argv) == 1:
+        url = "https://sni.velox.ch/"
+    else:
+        url = sys.argv[1]
+
+    # Check if it is a HTTPS website.
+    parts = urlparse.urlparse(url)
+
+    if parts.scheme.lower() != "https":
+        sys.stderr.write(
+            "Error: provided URL does not start with https://\n")
+        return 1
+
+    # Gather information
+    check_installation()
+
+    # Do the request.
+    if not can_import("requests"):
+        sys.stderr.exit("Error: cannot continue without requests module!\n")
+        return 1
+
+    sys.stdout.write("* Performing request: %s\n" % url)
+
+    import requests
+    requests.packages.urllib3.disable_warnings()
+
+    try:
+        try:
+            response = requests.get(url)
+        except requests.exceptions.SSLError as e:
+            sys.stdout.write(
+                "- Server certificate seems invalid. I will disable "
+                "certificate check and try again. You'll see the real "
+                "exception if it fails again.\n")
+            sys.stdout.write(
+                "* Retrying request with certificate verification off.\n")
+            response = requests.get(url)
+    except Exception as e:
+        sys.stdout.write(
+            "- An error occured while performing the request. The "
+            "exception was: %s\n" % e.message)
+        sys.stdout.write(
+            "- Consult the Troubleshooting wiki (https://github.com/"
+            "rembo10/headphones/wiki/Troubleshooting) before you post an "
+            "issue!")
+        return 0
+
+    # Verify the response.
+    if response.status_code == 200:
+        sys.stdout.write("+ Got a valid response. All seems OK!\n")
+    else:
+        sys.stdout.write(
+            "- Server returned status code %s. Expected a status code 200.\n",
+            response.status_code)
+        sys.stdout.write(
+            "- However, I was able to communicate to the server!\n")
+
+# E.g. `python sni_test.py https://example.org'.
+if __name__ == "__main__":
+    sys.exit(main())

data/interfaces/default/album.html

@@ -14,15 +14,15 @@
 
 			<a id="menu_link_delete" href="deleteAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}"><i class="fa fa-trash-o"></i> Delete Album</a>
 			%if album['Status'] == 'Skipped' or album['Status'] == 'Ignored':
-			<a id="menu_link_wanted" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true)" data-success="'${album['AlbumTitle']}' added to queue"><i class="fa fa-heart"></i> Mark Album as Wanted</a>
+			<a id="menu_link_wanted" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true)" data-success="'${album['AlbumTitle']}' added to queue"><i class="fa fa-heart"></i> Mark Album as Wanted</a>
 			%elif album['Status'] == 'Wanted':
-			<a id="menu_link_check" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this));" data-success="Forced checking successful"><i class="fa fa-search"></i> Force Check</a>
-			<a id="menu_link_skipped" href="#" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),true);" data-success="'${album['AlbumTitle']}' marked as Skipped"><i class="fa fa-step-forward"></i> Mark Album as Skipped</a>
+			<a id="menu_link_check" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this));" data-success="Forced checking successful"><i class="fa fa-search"></i> Force Check</a>
+			<a id="menu_link_skipped" href="javascript:void(0)" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),true);" data-success="'${album['AlbumTitle']}' marked as Skipped"><i class="fa fa-step-forward"></i> Mark Album as Skipped</a>
 			%else:
-			<a id="menu_link_retry" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true);" data-success="Retrying the same version of '${album['AlbumTitle']}'"><i class="fa fa-refresh"></i> Retry Download</a>
-			<a id="menu_link_new" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),true);" data-success="Looking for a new version of '${album['AlbumTitle']}'"><i class="fa fa-download"></i> Try New Version</a>
+			<a id="menu_link_retry" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=False', $(this),true);" data-success="Retrying the same version of '${album['AlbumTitle']}'"><i class="fa fa-refresh"></i> Retry Download</a>
+			<a id="menu_link_new" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),true);" data-success="Looking for a new version of '${album['AlbumTitle']}'"><i class="fa fa-download"></i> Try New Version</a>
 			%endif		
-			<a class="menu_link_edit" id="album_chooser" href="#"><i class="fa fa-pencil"></i> Choose Alternate Release</a>
+			<a class="menu_link_edit" id="album_chooser" href="javascript:void(0)"><i class="fa fa-pencil"></i> Choose Alternate Release</a>
 				<div id="dialog" title="Choose an Alternate Release" style="display:none" class="configtable">
 					<div class="links">
 						<%
@@ -30,11 +30,12 @@
 						%>
 						%if not alternate_albums:
 							<p>No alternate releases found. Try refreshing the artist (if the artist is being refreshed, please wait until it's finished)</p>
-							<h2><a id="refresh_artist" onclick="doAjaxCall('refreshArtist?ArtistID=${album['ArtistID']}', $(this)), true" href="#" data-success="'${album['ArtistName']}' is being refreshed">Refresh Artist</a></h2>
+							<h2><a id="refresh_artist" onclick="doAjaxCall('refreshArtist?ArtistID=${album['ArtistID']}', $(this)), true" href="javascript:void(0)" data-success="'${album['ArtistName']}' is being refreshed">Refresh Artist</a></h2>
 						%else:
 							%for alternate_album in alternate_albums:
 								<%
 									track_count = len(myDB.select("SELECT * from alltracks WHERE ReleaseID=?", [alternate_album['ReleaseID']]))
+									mb_link = "http://musicbrainz.org/release/" + alternate_album['ReleaseID']
 									have_track_count = len(myDB.select("SELECT * from alltracks WHERE ReleaseID=? AND Location IS NOT NULL", [alternate_album['ReleaseID']]))
 									if alternate_album['AlbumID'] == alternate_album['ReleaseID']:
 										alternate_album_name = "Headphones Default Release (" + str(alternate_album['ReleaseDate']) + ") [" + str(have_track_count) + "/" + str(track_count) + " tracks]"
@@ -42,50 +43,50 @@
 										alternate_album_name = alternate_album['AlbumTitle'] + " (" + alternate_album['ReleaseCountry'] + ", " + str(alternate_album['ReleaseDate']) + ", " + alternate_album['ReleaseFormat'] + ") [" + str(have_track_count) + "/" + str(track_count) + " tracks]"
 									
 								%>
-								<a href="#" onclick="doAjaxCall('switchAlbum?AlbumID=${album['AlbumID']}&ReleaseID=${alternate_album['ReleaseID']}', $(this), 'table');" data-success="Switched release to: ${alternate_album_name}">${alternate_album_name}</a><br>
+								<a href="javascript:void(0)" onclick="doAjaxCall('switchAlbum?AlbumID=${album['AlbumID']}&ReleaseID=${alternate_album['ReleaseID']}', $(this), 'table');" data-success="Switched release to: ${alternate_album_name}">${alternate_album_name}</a><a href="${mb_link}" target="_blank">MB</a><br>
 							%endfor
 						%endif
 					</div>
 				</div>
-			<a class="menu_link_edit" id="edit_search_term" href="#"><i class="fa fa-pencil"></i> Edit Search Term</a>
-				<div id="dialog2" title="Enter your own search term for this album" style="display:none" class="configtable">
-					<form action="editSearchTerm" method="GET" id="editSearchTerm">
-						<input type="hidden" name="AlbumID" value="${album['AlbumID']}">
-						<div class="row">
-							<%
-								if not album['SearchTerm']:
-									search_term = ""
-								else:
-									search_term = album['SearchTerm']
-								
-							%>
-							<input type="text" value="${search_term}" name="SearchTerm" size="40" />
-						</div>
-						<input type="button" value="Save changes" onclick="doAjaxCall('editSearchTerm',$(this),'tabs',true);return false;" data-success="Search term updated"/>
-					</form>
-				</div>	
-			<a class="menu_link_edit" id="choose_specific_download" href="#" onclick="getAvailableDownloads()"><i class="fa fa-search"></i> Choose Specific Download</a>
-				<div id="choose_specific_download_dialog" title="Choose a specific download for this album" style="display:none" class="configtable">
-						<table class="display" id="downloads_table">
-							<thead>
-								<tr>
-									<th id="title">Title</th>
-									<th id="size">Size</th>
-									<th id="provider">Provider</th>
-									<th id="kind">Kind</th>
-								</tr>
-							</thead>
-							<tbody id="downloads_table_body">
-							</tbody>
-						</table>
-				</div>	
+			<a class="menu_link_edit" id="edit_search_term" href="javascript:void(0)"><i class="fa fa-pencil"></i> Edit Search Term</a>
+			<div id="dialog2" title="Enter your own search term for this album" style="display:none" class="configtable">
+				<form action="editSearchTerm" method="GET" id="editSearchTerm">
+					<input type="hidden" name="AlbumID" value="${album['AlbumID']}">
+					<div class="row">
+						<%
+							if not album['SearchTerm']:
+								search_term = ""
+							else:
+								search_term = album['SearchTerm']
+						%>
+						<input type="text" value="${search_term}" name="SearchTerm" size="40" />
+					</div>
+					<input type="button" value="Save changes" onclick="doAjaxCall('editSearchTerm',$(this),'tabs',true);return false;" data-success="Search term updated"/>
+				</form>
+			</div>
+			<a class="menu_link_edit" id="choose_specific_download" href="javascript:void(0)" onclick="getAvailableDownloads()"><i class="fa fa-search"></i> Choose Specific Download</a>
+			<div id="choose_specific_download_dialog" title="Choose a specific download for this album" style="display:none" class="configtable">
+				<table class="display" id="downloads_table">
+					<thead>
+						<tr>
+							<th id="title">Title</th>
+							<th id="size">Size</th>
+							<th id="provider">Provider</th>
+							<th id="kind">Kind</th>
+							<th id="matches" title="Matches quality and/or seed settings.">Matches</th>
+						</tr>
+					</thead>
+					<tbody id="downloads_table_body">
+					</tbody>
+				</table>
+			</div>
 		</div>
-	</div>	
+	</div>
 </%def>
 
 <%def name="body()">
 	<div class="table_wrapper">
-		<div id="albumheader" class="clearfix">	
+		<div id="albumheader" class="clearfix">
 			<div id="albumImg">
 				<img height="200" alt="" class="albumArt" src="artwork/album/${album['AlbumID']}">
 			</div>
@@ -197,14 +198,20 @@
 		var elem = $("#albumInfo");
 		getInfo(elem,id,'album');
 	}
-	
+
 	function initThisPage() {
 		$('#album_chooser').click(function() {
-			$('#dialog').dialog({ width: "500px" });
+			$('#dialog').dialog({
+				width: 500,
+				maxHeight: 500
+			});
 			return false;
 		});
 		$('#edit_search_term').click(function() {
-			$('#dialog2').dialog({ width: "500px" });
+			$('#dialog2').dialog({
+				width: 500,
+				maxHeight: 500
+			});
 			return false;
 		});
 		$('#refresh_artist').click(function() {
@@ -213,8 +220,8 @@
 		initActions();
 		setTimeout(function(){
 			initFancybox();
-			},1000);
-		
+		}, 1000);
+
 		$('#track_table').dataTable({
 			"aaSorting": [],
 			"bFilter": false,
@@ -231,21 +238,26 @@
 			feedback.fadeOut();
 			search_results = data
 			for( var i = 0, len = data.length; i < len; i++ ) {
-    			$('#downloads_table_body').append('<tr><td id="title"><a href="#" onclick="downloadSpecificRelease('+i+')">'+data[i].title+'</a></td><td id="size"><span title='+data[i].size+'></span>'+(data[i].size / (1024*1024)).toFixed(2)+' MB</td><td id="provider">'+data[i].provider+'</td><td id="kind">'+data[i].kind+'</td></tr>');
+    			$('#downloads_table_body').append('<tr><td id="title"><a href="javascript:void(0)" onclick="downloadSpecificRelease('+i+')">'+data[i].title+'</a></td><td id="size"><span title='+data[i].size+'></span>'+(data[i].size / (1024*1024)).toFixed(2)+' MB</td><td id="provider">'+data[i].provider+'</td><td id="kind">'+data[i].kind+'</td><td id="matches">'+data[i].matches+'</td></tr>');
     		}
 			$('#downloads_table').dataTable({
 				"aoColumns": [
 					null,
-					{ "sType": "title-numeric"},
+					{"sType": "title-numeric"},
 					null,
-					null
+					null,
+					{"sType": "string"}
 				],
-				"aaSorting": [],
+				"aaSorting": [[ 4, 'desc']],
 				"bFilter": false,
 				"bInfo": false,
 				"bPaginate": false,
+				"bDestroy": true
 			});
-    		$("#choose_specific_download_dialog").dialog({ width: "1000px" });
+    		$("#choose_specific_download_dialog").dialog({
+    			width: "80%",
+    			maxHeight: 500
+    		});
     		return false;
 		});
 	}
@@ -326,6 +338,21 @@
         });
     }
 
+    jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+		"title-numeric-pre": function ( a ) {
+			var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+			return parseFloat( x );
+		},
+
+		"title-numeric-asc": function ( a, b ) {
+			return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+		},
+
+        "title-numeric-desc": function ( a, b ) {
+            return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+        }
+    } );
+
 	$(document).ready(function()	{
 		getAlbumInfo();
 		initThisPage();

data/interfaces/default/artist.html

@@ -8,18 +8,19 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a id="menu_link_refresh" onclick="doSimpleAjaxCall('refreshArtist?ArtistID=${artist['ArtistID']}')" href="#"><i class="fa fa-refresh"></i> Refresh Artist</a>
+			<a id="menu_link_refresh" onclick="doSimpleAjaxCall('refreshArtist?ArtistID=${artist['ArtistID']}')" href="javascript:void(0)"><i class="fa fa-refresh"></i> Refresh Artist</a>
 			<a id="menu_link_delete" href="deleteArtist?ArtistID=${artist['ArtistID']}"><i class="fa fa-trash-o"></i> Delete Artist</a>
+			<a id="menu_link_scan" onclick="doAjaxCall('scanArtist?ArtistID=${artist['ArtistID']}', $(this)),'table'" href="javascript:void(0)" data-success="'${artist['ArtistName']}' was scanned"><i class="fa fa-refresh"></i> Scan Artist</a>
 			%if artist['Status'] == 'Paused':
-				<a id="menu_link_resume" href="#" onclick="doAjaxCall('resumeArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} resumed"><i class="fa fa-play"></i> Resume Artist</a>
+				<a id="menu_link_resume" href="javascript:void(0)" onclick="doAjaxCall('resumeArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} resumed"><i class="fa fa-play"></i> Resume Artist</a>
 			%else:
-				<a id="menu_link_pauze" href="#" onclick="doAjaxCall('pauseArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} paused"><i class="fa fa-pause"></i> Pause Artist</a>
+				<a id="menu_link_pauze" href="javascript:void(0)" onclick="doAjaxCall('pauseArtist?ArtistID=${artist['ArtistID']}',$(this),true)" data-success="${artist['ArtistName']} paused"><i class="fa fa-pause"></i> Pause Artist</a>
 			%endif
 			%if artist['IncludeExtras']:
-				<a id="menu_link_removeextra" href="#" onclick="doAjaxCall('removeExtras?ArtistID=${artist['ArtistID']}&ArtistName=${artist['ArtistName']}',$(this),'submenu&table')" data-success="Extras removed for ${artist['ArtistName']}"><i class="fa fa-minus"></i> Remove Extras</a>
-				<a class="menu_link_edit" id="menu_link_modifyextra" href="#"><i class="fa fa-pencil"></i> Modify Extras</a>
+				<a id="menu_link_removeextra" href="javascript:void(0)" onclick="doAjaxCall('removeExtras?ArtistID=${artist['ArtistID']}&ArtistName=${artist['ArtistName']}',$(this),'submenu&table')" data-success="Extras removed for ${artist['ArtistName']}"><i class="fa fa-minus"></i> Remove Extras</a>
+				<a class="menu_link_edit" id="menu_link_modifyextra" href="javascript:void(0)"><i class="fa fa-pencil"></i> Modify Extras</a>
 			%else:
-				<a id="menu_link_getextra" href="#"><i class="fa fa-plus"></i> Get Extras</a>
+				<a id="menu_link_getextra" href="javascript:void(0)"><i class="fa fa-plus"></i> Get Extras</a>
 			%endif
 				<div id="dialog" title="Choose Which Extras to Fetch" style="display:none" class="configtable">
 					<form action="getExtras" method="get" class="form">
@@ -69,6 +70,7 @@
 				<th id="albumname">Name</th>
 				<th id="reldate">Date</th>
 				<th id="type">Type</th>
+				<th id="score">Metacritic</th>
 				<th id="status">Status</th>
 				<th id="have">Have</th>
 				<th id="bitrate">Bitrate</th>
@@ -124,18 +126,19 @@
 				<td id="albumname"><a href="albumPage?AlbumID=${album['AlbumID']}">${album['AlbumTitle']}</a></td>
 				<td id="reldate">${album['ReleaseDate']}</td>
 				<td id="type">${album['Type']}</td>
+				<td id="score">${album['CriticScore']}/${album['UserScore']}</td>
 				<td id="status">${album['Status']}
 				%if album['Status'] == 'Skipped' or album['Status'] == 'Ignored':
-					[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' added to Wanted list">want</a>]
+					[<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' added to Wanted list">want</a>]
 				%elif (album['Status'] == 'Wanted' or album['Status'] == 'Wanted Lossless'):
-                    [<a href="#" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' skipped">skip</a>] [<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Trying to download'${album['AlbumTitle']}'" title="Search if available for download">search</a>]
+                    [<a href="javascript:void(0)" onclick="doAjaxCall('unqueueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}',$(this),'table')" data-success="'${album['AlbumTitle']}' skipped">skip</a>] [<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Trying to download'${album['AlbumTitle']}'" title="Search if available for download">search</a>]
 				%else:
-					[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Retrying the same version of '${album['AlbumTitle']}'" title="Retry the same download again">retry</a>][<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),'table')" title="Try a new download, skipping all previously tried nzbs"  data-success="Downloading new version for '${album['AlbumTitle']}'" data-success="Looking for a new version of '${album['AlbumTitle']}'">new</a>]
+					[<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}', $(this),'table')" data-success="Retrying the same version of '${album['AlbumTitle']}'" title="Retry the same download again">retry</a>][<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&new=True', $(this),'table')" title="Try a new download, skipping all previously tried nzbs"  data-success="Downloading new version for '${album['AlbumTitle']}'" data-success="Looking for a new version of '${album['AlbumTitle']}'">new</a>]
 				%endif
 				%if albumformat in lossy_formats and album['Status'] == 'Skipped':
-					[<a id="wantlossless" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Lossless version of '${album['AlbumTitle']}' added to queue">want lossless</a>]
+					[<a id="wantlossless" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Lossless version of '${album['AlbumTitle']}' added to queue">want lossless</a>]
 				%elif albumformat in lossy_formats and (album['Status'] == 'Snatched' or album['Status'] == 'Downloaded'):
-					[<a id="wantlossless" href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Retrying the same lossless version of '${album['AlbumTitle']}'">retry lossless</a>]
+					[<a id="wantlossless" href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${album['AlbumID']}&ArtistID=${album['ArtistID']}&lossless=True', $(this),'table')" data-success="Retrying the same lossless version of '${album['AlbumTitle']}'">retry lossless</a>]
 				%endif
 				</td>
 				<td id="have"><span title="${percent}"><span><div class="progress-container"><div style="width:${percent}%"><div class="havetracks">${havetracks}/${totaltracks}</div></div></div></td>
@@ -163,21 +166,21 @@
 	}
 
 	<%
-    if headphones.SONGKICK_FILTER_ENABLED:
+    if headphones.CONFIG.SONGKICK_FILTER_ENABLED:
         songkick_filter_enabled = "true"
     else:
         songkick_filter_enabled = "false"
 
-    if not headphones.SONGKICK_LOCATION:
+    if not headphones.CONFIG.SONGKICK_LOCATION:
     	songkick_location = "none"
     else:
-        songkick_location = headphones.SONGKICK_LOCATION
-    	
-    if headphones.SONGKICK_ENABLED:
+        songkick_location = headphones.CONFIG.SONGKICK_LOCATION
+
+    if headphones.CONFIG.SONGKICK_ENABLED:
         songkick_enabled = "true"
     else:
         songkick_enabled = "false"
-	
+
 	%>
 	function getArtistsCalendar() {
 		var template, calendarDomNode;
@@ -186,48 +189,48 @@
 
 		template = '<li><a target="_blank" href="URI"><span class="sk-name">NAME</span><span class="sk-location">LOC</span></a></li>';
 
-		$.getJSON("https://api.songkick.com/api/3.0/artists/mbid:${artist['ArtistID']}/calendar.json?apikey=${headphones.SONGKICK_APIKEY}&jsoncallback=?",
+		$.getJSON("https://api.songkick.com/api/3.0/artists/mbid:${artist['ArtistID']}/calendar.json?apikey=${headphones.CONFIG.SONGKICK_APIKEY}&jsoncallback=?",
 			function(data){
-				if (data['resultsPage'].totalEntries >= 1) {          
-          
-                    if( ${songkick_filter_enabled} ) {
+				if (data['resultsPage'].totalEntries >= 1) {
+                    if (${songkick_filter_enabled}) {
                         data.resultsPage.results.event = $.grep(data.resultsPage.results.event, function(element,index){
-                        return element.venue.metroArea.id == ${songkick_location};
-                    });
+                    	    return element.venue.metroArea.id == ${songkick_location};
+                    	});
                     }
-          
-					var tourDate;
 
-					calendarDomNode.show();
-					$("#artistImg").addClass('on-tour');
+                    if (data.resultsPage.results.event.length > 0) {
+						var tourDate;
 
-					jQuery.each( data['resultsPage'].results.event, function( i, event ) {
-						tourDate = template;
-						tourDate = tourDate.replace('URI',event.uri);
-						tourDate = tourDate.replace('NAME',event.displayName);
-						tourDate = tourDate.replace('LOC',event.location.city);
+						calendarDomNode.show();
+						$("#artistImg").addClass('on-tour');
 
-						calendarDomNode.append(tourDate);
-					});
+						jQuery.each(data.resultsPage.results.event, function(i, event) {
+							tourDate = template;
+							tourDate = tourDate.replace('URI',event.uri);
+							tourDate = tourDate.replace('NAME',event.displayName);
+							tourDate = tourDate.replace('LOC',event.location.city);
 
-					calendarDomNode.append('<li><img src="interfaces/default/images/songkick.png" alt="concerts by songkick" class="sk-logo" /></li>');
-					
-				$(function() {
-				  $("#artistCalendar").each(function() {
-					$("li:gt(4)", this).hide(); /* :gt() is zero-indexed */
-					$("li:nth-child(5)", this).after("<br><li class='more'><a href='#'>More...</a></li>"); /* :nth-child() is one-indexed */
-				  });
-				  $("li.more a").live("click", function() {
-					var li = $(this).parents("li:first");
-					li.parent().children().show();
-					li.remove();
-					return false;
-				  });
-				});
+							calendarDomNode.append(tourDate);
+						});
 
+						calendarDomNode.append('<li><img src="interfaces/default/images/songkick.png" alt="concerts by songkick" class="sk-logo" /></li>');
+
+						$(function() {
+						  $("#artistCalendar").each(function() {
+							$("li:gt(4)", this).hide(); /* :gt() is zero-indexed */
+							$("li:nth-child(5)", this).after("<br><li class='more'><a href='#'>More...</a></li>"); /* :nth-child() is one-indexed */
+						  });
+						  $("li.more").on("click", 'a', function() {
+							var li = $(this).parents("li:first");
+							li.parent().children().show();
+							li.remove();
+							return false;
+						  });
+						});
+					}
 				}
-			// data is JSON response object
-		});
+			}
+		);
 	}
 
 	var loadingMessage = false;
@@ -269,7 +272,7 @@
 			$('#dialog').dialog();
 			event.preventDefault();
 		});
-		$('#menu_link_modifyextra').click(function() {
+		$('#menu_link_modifyextra').click(function(event) {
 			$('#dialog').dialog();
 			event.preventDefault();
 		});
@@ -282,6 +285,7 @@
 				{ "sType": "date" },
 				null,
 				null,
+				null,
 				{ "sType": "title-numeric"},
 				null,
 				null

data/interfaces/default/base.html

@@ -31,14 +31,14 @@
 <body>
 	<div id="container">
 		<div id="ajaxMsg" class="ajaxMsg"></div>
-		% if not headphones.CURRENT_VERSION:
+		% if headphones.CONFIG.CHECK_GITHUB and not headphones.CURRENT_VERSION:
 			<div id="updatebar">
 				You're running an unknown version of Headphones. <a href="update">Update</a> or
-				<a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
+				<a href="javascript:void(0)" onclick="$('#updatebar').slideUp('slow');">Close</a>
 			</div>
-		% elif headphones.CURRENT_VERSION != headphones.LATEST_VERSION and headphones.COMMITS_BEHIND > 0 and headphones.INSTALL_TYPE != 'win':
+		% elif headphones.CONFIG.CHECK_GITHUB and headphones.CURRENT_VERSION != headphones.LATEST_VERSION and headphones.COMMITS_BEHIND > 0 and headphones.INSTALL_TYPE != 'win':
 			<div id="updatebar">
-				A <a href="https://github.com/${headphones.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="#" onclick="$('#updatebar').slideUp('slow');">Close</a>
+				A <a href="https://github.com/${headphones.CONFIG.GIT_USER}/headphones/compare/${headphones.CURRENT_VERSION}...${headphones.LATEST_VERSION}"> newer version</a> is available. You're ${headphones.COMMITS_BEHIND} commits behind. <a href="update">Update</a> or <a href="javascript:void(0)" onclick="$('#updatebar').slideUp('slow');">Close</a>
 			</div>
 		% endif
 
@@ -52,7 +52,7 @@
 				<li><a href="extras">extras</a></li>
 				<li><a href="manage">manage</a></li>
 				<li><a href="history">history</a></li>
-				<li><a href="logs" class="log">logs</a></li>
+				<li><a href="logs">logs</a></li>
 				<li><a href="config" class="config"><i class="fa fa-gear fa-lg"></i></a></li>
 			</ul>
 			<div id="searchbar">
@@ -62,6 +62,7 @@
 					<select name="type" id="search_type">
 						<option value="artist">Artist</option>
 						<option value="album">Album</option>
+						<option value="series">Series</option>
 					</select>
 					<input type="submit" value="Search"/>
 				</form>
@@ -88,7 +89,7 @@
 				<small>
 				<a href="shutdown"><i class="fa fa-power-off"></i> Shutdown</a> |
 				<a href="restart"><i class="fa fa-power-off"></i> Restart</a> |
-				<a href="#" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh"></i> Check for new version</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh"></i> Check for new version</a>
 				</small>
 			</div>
 			<div id="version">
@@ -96,8 +97,8 @@
 				%if version.HEADPHONES_VERSION != 'master':
 				(${version.HEADPHONES_VERSION})
 				%endif
-				%if headphones.GIT_BRANCH != 'master':
-				(${headphones.GIT_BRANCH})
+				%if headphones.CONFIG.GIT_BRANCH != 'master':
+				(${headphones.CONFIG.GIT_BRANCH})
 				%endif
 			</div>
 		</footer>
@@ -119,7 +120,7 @@
 	    $(document).ready(function() {
 	        $('form:first *:input[type!=hidden]:first').focus();
 	        try{
-	            var type = window.localStorage.getItem('search_type');
+	            var type = window.localStorage.getItem('search_type') || "artist";
 	            $("#search_type").val(type);
 	        } catch(e) {
 	        }
@@ -139,4 +140,4 @@
 
 <%def name="javascriptIncludes()"></%def>
 <%def name="headIncludes()"></%def>
-<%def name="headerIncludes()"></%def>
+<%def name="headerIncludes()"></%def>

data/interfaces/default/css/style.css

@@ -325,7 +325,7 @@ form fieldset small.heading {
   margin-top: -15px;
 }
 form .row {
-  font-family: Helvetica, Arial;
+  font-family: Helvetica, Arial, sans-serif;
   margin-bottom: 10px;
 }
 form .row label {
@@ -334,7 +334,11 @@ form .row label {
   font-size: 12px;
   line-height: normal;
   padding-top: 7px;
-  width: 175px;
+  width: 170px;
+}
+form .row label.inline {
+  margin-right: 5px;
+  width: auto;
 }
 form .row input {
   margin-right: 5px;
@@ -364,7 +368,7 @@ form .row small {
   display: block;
   font-size: 9px;
   line-height: 12px;
-  margin-left: 175px;
+  margin-left: 170px;
   margin-top: 3px;
 }
 form .left label {
@@ -404,6 +408,18 @@ form .checkbox small {
 form .indent input {
   margin-left: 15px;
 }
+form .suboptions {
+  margin-left: 15px;
+}
+.option{
+  font-size: 15px;
+  font-weight: 600;
+  vertical-align: middle;
+}
+input.bigcheck[type="checkbox"] {
+  width: 16px;
+  height: 16px;
+}
 ul,
 ol {
   margin-left: 2em;
@@ -1416,6 +1432,13 @@ div#artistheader h2 a {
   line-height: 10px;
   height: 10px;
 }
+.nopad {
+  margin-bottom: 0px !important;
+  margin-top: 0px !important;
+  padding-top: 0px !important;
+  padding-bottom: 0px !important;
+  padding: 0px 0px !important;
+}
 #album_table th#albumname,
 #upcoming_table th#artistname,
 #wanted_table th#artistname {
@@ -1441,6 +1464,7 @@ div#artistheader h2 a {
 #album_table td#albumname,
 #album_table td#reldate,
 #album_table td#type,
+#album_table td#score,
 #track_table td#duration,
 #upcoming_table td#select,
 #upcoming_table td#status,
@@ -1451,6 +1475,7 @@ div#artistheader h2 a {
 }
 #album_table td#status,
 #album_table td#bitrate,
+#album_table td#score,
 #album_table td#albumformat,
 #album_table td#wantlossless {
   font-size: 13px;

data/interfaces/default/css/style.less

@@ -190,7 +190,12 @@ form {
             font-size: 12px;
             line-height: normal;
             padding-top: 7px;
-            width: 175px;
+            width: 170px;
+
+            &.inline {
+                margin-right: 5px;
+                width: auto;
+            }
         }
         input { margin-right: 5px; }
         input[type=text], input[type=password] {
@@ -211,7 +216,7 @@ form {
             display: block;
             font-size: 9px;
             line-height: 12px;
-            margin-left: 175px;
+            margin-left: 170px;
             margin-top: 3px;
         }
     }

data/interfaces/default/history.html

@@ -7,12 +7,13 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=all',$(this),'table')" data-success="All History cleared"><i class="fa fa-trash-o"></i> Clear All History</a>
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Processed',$(this),'table')" data-success="All Processed cleared"><i class="fa fa-trash-o"></i> Clear Processed</a>
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Unprocessed',$(this),'table')" data-success="All Unprocessed cleared"><i class="fa fa-trash-o"></i> Clear Unprocessed</a>
-			<a id="menu_link_delete" href="#" onclick="doAjaxCall('clearhistory?type=Snatched',$(this),'table')" data-success="All Snatched cleared"><i class="fa fa-trash-o"></i> Clear Snatched</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=all',$(this),'table')" data-success="All History cleared"><i class="fa fa-trash-o"></i> Clear All History</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Processed',$(this),'table')" data-success="All Processed cleared"><i class="fa fa-trash-o"></i> Clear Processed</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Unprocessed',$(this),'table')" data-success="All Unprocessed cleared"><i class="fa fa-trash-o"></i> Clear Unprocessed</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Frozen',$(this),'table')" data-success="All Frozen cleared"><i class="fa fa-trash-o"></i> Clear Frozen</a>
+			<a id="menu_link_delete" href="javascript:void(0)" onclick="doAjaxCall('clearhistory?type=Snatched',$(this),'table')" data-success="All Snatched cleared"><i class="fa fa-trash-o"></i> Clear Snatched</a>
 		</div>
-	</div>	
+	</div>
 </%def>
 
 <%def name="body()">
@@ -31,32 +32,41 @@
 			</tr>
 		</thead>
 		<tbody>
-		%for item in history:
-			<%
-				if item['Status'] == 'Processed':
-					grade = 'A'
-				elif item['Status'] == 'Snatched':
-					grade = 'C'
-				elif item['Status'] == 'Unprocessed':
-					grade = 'X'
-				else:
-					grade = 'U'
-					
-				fileid = 'unknown'
-				if item['URL'].find('nzb') != -1:
-					fileid = 'nzb'
-				if item['URL'].find('torrent') != -1:
-					fileid = 'torrent'
-				if item['URL'].find('rutracker') != -1:
-					fileid = 'torrent'
-			%>
+        %for item in history:
+        <%
+            if item['Status'] == 'Processed':
+                grade = 'A'
+            elif item['Status'] == 'Snatched':
+                grade = 'C'
+            elif item['Status'] == 'Unprocessed':
+                grade = 'X'
+            elif item['Status'] == 'Frozen':
+                grade = 'X'
+            else:
+                grade = 'U'
+
+            fileid = 'unknown'
+            if item['URL'].find('nzb') != -1:
+                fileid = 'nzb'
+            if item['URL'].find('torrent') != -1:
+                fileid = 'torrent'
+            if item['URL'].find('magnet:') != -1:
+                fileid = 'torrent'
+            if item['URL'].find('rutracker') != -1:
+                fileid = 'torrent'
+            if item['URL'].find('codeshy') != -1:
+                fileid = 'nzb'
+
+            folder = 'Folder: ' + item['FolderName']
+
+        %>
 			<tr class="grade${grade}">
 				<td id="dateadded">${item['DateAdded']}</td>
 				<td id="filename">${cgi.escape(item['Title'], quote=True)} [<a href="${item['URL']}">${fileid}</a>]<a href="albumPage?AlbumID=${item['AlbumID']}">[album page]</a></td>
 				<td id="size">${helpers.bytes_to_mb(item['Size'])}</td>
-				<td id="status">${item['Status']}</td>
-				<td id="action">[<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&redirect=history', $(this),'table')"  data-success="Retrying download of '${cgi.escape(item['Title'], quote=True)}'">retry</a>][<a href="#" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&new=True&redirect=history',$(this),'table')"  data-success="Looking for a new version of '${cgi.escape(item['Title'], quote=True)}'">new</a>]</td>
-				<td id="delete"><a href="#" onclick="doAjaxCall('clearhistory?date_added=${item['DateAdded']}&title=${cgi.escape(item['Title'], quote=True)}',$(this),'table')" data-success="${cgi.escape(item['Title'], quote=True)} cleared from history"><img src="interfaces/default/images/trashcan.png" height="18" width="18" id="trashcan" title="Clear this item from the history"></a>
+				<td title="${folder}" id="status">${item['Status']}</td>
+				<td id="action">[<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&redirect=history', $(this),'table')"  data-success="Retrying download of '${cgi.escape(item['Title'], quote=True)}'">retry</a>][<a href="javascript:void(0)" onclick="doAjaxCall('queueAlbum?AlbumID=${item['AlbumID']}&new=True&redirect=history',$(this),'table')"  data-success="Looking for a new version of '${cgi.escape(item['Title'], quote=True)}'">new</a>]</td>
+				<td id="delete"><a href="javascript:void(0)" onclick="doAjaxCall('clearhistory?date_added=${item['DateAdded']}&title=${cgi.escape(item['Title'], quote=True)}',$(this),'table')" data-success="${cgi.escape(item['Title'], quote=True)} cleared from history"><img src="interfaces/default/images/trashcan.png" height="18" width="18" id="trashcan" title="Clear this item from the history"></a>
 			</tr>
 		%endfor
 		</tbody>
@@ -81,8 +91,11 @@
 				"sInfoFiltered":"(filtered from _MAX_ total items)"},
 			"iDisplayLength": 25,
 			"sPaginationType": "full_numbers",
-			"aaSorting": []
-
+			"aaSorting": [],
+			"fnDrawCallback": function (o) {
+                // Jump to top of page
+                $('html,body').scrollTop(0);
+            }
 		});
 		resetFilters("history");
 	}

data/interfaces/default/index.html

@@ -10,7 +10,7 @@
 				<th id="albumart"></th>
 				<th id="name">Artist Name</th>
 				<th id="status">Status</th>
-				<th id="album">Latest Album</th>
+				<th id="album">Latest Release</th>
 				<th id="have">Have</th>
 			</tr>
 		</thead>
@@ -131,6 +131,10 @@
                     },
                 "fnInitComplete": function(oSettings, json)
                 {
+                },
+                "fnDrawCallback": function (o) {
+                    // Jump to top of page
+                    $('html,body').scrollTop(0);
                 }
             });
             $('#artist_table').on("draw.dt", function () {

data/interfaces/default/logs.html

@@ -7,6 +7,7 @@
     <div id="subhead_container">
         <div id="subhead_menu">
             <a class="menu_link_edit" href="clearLogs"><i class="fa fa-trash-o"></i> Clear log</a>
+            <a class="menu_link_edit" href="toggleVerbose"><i class="fa fa-pencil"></i> Toggle Debug Log</a>
         </div>
     </div>
 </%def>
@@ -46,51 +47,50 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-$(document).ready(function() {
-    initActions();
+        $(document).ready(function() {
+            initActions();
 
-	$('#log_table').dataTable( {
-		"bProcessing": true,
-        "bServerSide": true,
-		"sAjaxSource": 'getLog',
-        "sPaginationType": "full_numbers",
-        "aaSorting": [[0, 'desc']],
-        "iDisplayLength": 25,
-        "bStateSave": true,
-        "oLanguage": {
-                    "sSearch":"Filter:",
-					"sLengthMenu":"Show _MENU_ lines per page",
-					"sEmptyTable": "No log information available",
-					"sInfo":"Showing _START_ to _END_ of _TOTAL_ lines",
-					"sInfoEmpty":"Showing 0 to 0 of 0 lines",
-					"sInfoFiltered":"(filtered from _MAX_ total lines)"},
-        "fnRowCallback": function (nRow, aData, iDisplayIndex, iDisplayIndexFull) {
-                                if (aData[1] === "ERROR")
-                                {
-                                    $('td', nRow).closest('tr').addClass("gradeX");
-                                }
-                                else if (aData[1] === "WARNING")
-                                {
-                                    $('td', nRow).closest('tr').addClass("gradeW");
-                                }
-                                else
-                                {
-                                    $('td', nRow).closest('tr').addClass("gradeZ");
-                                }
+        	$('#log_table').dataTable( {
+        		"bProcessing": true,
+                "bServerSide": true,
+        		"sAjaxSource": 'getLog',
+                "sPaginationType": "full_numbers",
+                "aaSorting": [[0, 'desc']],
+                "iDisplayLength": 25,
+                "bStateSave": true,
+                "oLanguage": {
+                            "sSearch":"Filter:",
+        					"sLengthMenu":"Show _MENU_ lines per page",
+        					"sEmptyTable": "No log information available",
+        					"sInfo":"Showing _START_ to _END_ of _TOTAL_ lines",
+        					"sInfoEmpty":"Showing 0 to 0 of 0 lines",
+        					"sInfoFiltered":"(filtered from _MAX_ total lines)"},
+                "fnRowCallback": function (nRow, aData, iDisplayIndex, iDisplayIndexFull) {
+                    if (aData[1] === "ERROR") {
+                        $('td', nRow).closest('tr').addClass("gradeX");
+                    } else if (aData[1] === "WARNING") {
+                        $('td', nRow).closest('tr').addClass("gradeW");
+                    } else {
+                        $('td', nRow).closest('tr').addClass("gradeZ");
+                    }
 
-                                return nRow;
-                                },
-        "fnServerData": function ( sSource, aoData, fnCallback ) {
-			/* Add some extra data to the sender */
-			$.getJSON( sSource, aoData, function (json) { 
-				fnCallback(json)
-			} );
-		}
-	} );
-} );
+                    return nRow;
+                },
+                "fnDrawCallback": function (o) {
+                    // Jump to top of page
+                    $('html,body').scrollTop(0);
+                },
+                "fnServerData": function ( sSource, aoData, fnCallback ) {
+        			/* Add some extra data to the sender */
+        			$.getJSON(sSource, aoData, function (json) {
+        				fnCallback(json)
+        			});
+        		}
+        	});
+        });
 	</script>
 	<script>
-		var timer; 
+		var timer;
 		function setRefresh()
 		{
 			refreshrate = document.getElementById('refreshrate');

data/interfaces/default/manage.html

@@ -6,7 +6,7 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a class="menu_link_edit" id="manage_albums" href="#"><i class="fa fa-pencil"></i> Manage Albums</a>
+			<a class="menu_link_edit" id="manage_albums" href="javascript:void(0)"><i class="fa fa-pencil"></i> Manage Albums</a>
 				<div id="dialog" title="Choose Album Filter" style="display:none" class="configtable">
 					<div class="links">
 						<a href="manageAlbums?Status=Downloaded"><i class="fa fa-check fa-fw"></i> Manage Downloaded Albums</a><br>
@@ -20,7 +20,7 @@
 					</div>
 				</div>
 			<a class="menu_link_edit" href="manageArtists"><i class="fa fa-pencil"></i> Manage Artists</a>
-			%if not headphones.ADD_ARTISTS:
+			%if not headphones.CONFIG.AUTO_ADD_ARTISTS:
 			<a class="menu_link_edit" href="manageNew"><i class="fa fa-pencil"></i> Manage New Artists</a>
 			%endif
 			<a class="menu_link_edit" href="manageUnmatched"><i class="fa fa-pencil"></i> Manage Unmatched</a>
@@ -53,18 +53,18 @@
 					<br/>
 					<div class="row">
 								<label for="">Path to directory</label>
-					%if headphones.MUSIC_DIR:
-						<input type="text" value="${headphones.MUSIC_DIR}" name="path" size="70" />
+					%if headphones.CONFIG.MUSIC_DIR:
+						<input type="text" value="${headphones.CONFIG.MUSIC_DIR}" name="path" size="70" />
 					%else:
 						<input type="text" value="Enter a Music Directory to scan" onfocus="if
 						(this.value==this.defaultValue) this.value='';" name="path" size="70" />
 					%endif
 					</div>
 					<div class="row checkbox"> 
-					<input type="checkbox" name="libraryscan" id="libraryscan" value="1" ${checked(headphones.LIBRARYSCAN)}><label>Automatically scan library</label>
+					<input type="checkbox" name="libraryscan" id="libraryscan" value="1" ${checked(headphones.CONFIG.LIBRARYSCAN)}><label>Automatically scan library</label>
 					</div>
 					<div class="row checkbox"> 
-					<input type="checkbox" name="autoadd" id="autoadd" value="1" ${checked(headphones.ADD_ARTISTS)}><label>Auto-add new artists</label>
+					<input type="checkbox" name="autoadd" id="autoadd" value="1" ${checked(headphones.CONFIG.AUTO_ADD_ARTISTS)}><label>Auto-add new artists</label>
 					</div>
 								
 				</fieldset>
@@ -83,14 +83,14 @@
 			<div class="row">
 				<label for="">Username</label>
 				<%
-					if headphones.LASTFM_USERNAME:
-						lastfmvalue = headphones.LASTFM_USERNAME
+					if headphones.CONFIG.LASTFM_USERNAME:
+						lastfmvalue = headphones.CONFIG.LASTFM_USERNAME
 					else:
 						lastfmvalue = ''
 				%>
 				<input type="text" value="${lastfmvalue}" placeholder="Last.fm username" onfocus="if
 				(this.value==this.defaultValue) this.value='';" name="username" id="username" size="18" />
-				<a href="#" onclick="doAjaxCall('importLastFM?username=',$(this),'tabs');return false;" data-success="Last.fm username has been reset"><i class="fa fa-reply"></i> Reset username</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('importLastFM?username=',$(this),'tabs');return false;" data-success="Last.fm username has been reset"><i class="fa fa-reply"></i> Reset username</a>
 			</div>
 			</fieldset>
 			<input type="button" value="Save changes" onclick="doAjaxCall('importLastFM',$(this),'tabs',true);return false;" data-success="Last.fm artists will be imported" data-error="Fill in a last.fm username"/>
@@ -121,10 +121,10 @@
 		<fieldset>
 			<legend>Force Search</legend>
 			<div class="links">
-				<a href="#" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search fa-fw"></i> Force Check for Wanted Albums</a>
-				<a href="#" onclick="doAjaxCall('forceUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Fast]</a>
-				<a href="#" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh fa-fw"></i> Check for Headphones Updates</a>
-				<a href="#" id="delete_empty_artists"><i class="fa fa-trash-o fa-fw"></i> Delete empty Artists</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search fa-fw"></i> Force Check for Wanted Albums</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Fast]</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('checkGithub',$(this))" data-success="Checking for update successful" data-error="Error checking for update"><i class="fa fa-refresh fa-fw"></i> Check for Headphones Updates</a>
+				<a href="javascript:void(0)" id="delete_empty_artists"><i class="fa fa-trash-o fa-fw"></i> Delete empty Artists</a>
 					<div id="emptyartistdialog" title="Confirm Artist Deletion" style="display:none" class="configtable">
 					    %if emptyArtists:
 						<h3>The following artists will be deleted:</h3>
@@ -137,28 +137,27 @@
 					        No empty artists found.
 					    %endif
 					</div>
-				<a href="#" onclick="doAjaxCall('forcePostProcess',$(this))" data-success="Post-Processor is being loaded" data-error="Error during Post-Processing"><i class="fa fa-wrench fa-fw"></i> Force Post-Process Albums in Download Folder</a>
+					<div id="post_process">
+					<a href="javascript:void(0)" class="btnOpenDialog"><i class="fa fa-wrench fa-fw"></i> Force Post-Process Albums in Download Folder</a>
+					</div>
 			</div>
 		</fieldset>
-		<form action="forcePostProcess" method="GET">
-		            <fieldset>
-		            <div class="row">
-		                <label>Force Post-Process Albums in Alternate Folder</label>
-					    <input type="text" value="" name="dir" id="dir" size="50" /><input type="submit" />
-		            </div>
-		            </fieldset>
-		      
-		</form>
-		<form action="forcePostProcess" method="GET">
-		            <fieldset>
-		            <div class="row">
-		                <label>Post-Process Single Folder</label>
-					    <input type="text" value="" name="album_dir" id="album_dir" size="50" /><input type="submit" />
-		            </div>
-		            </fieldset>
-		      
-		</form>
-		
+		<fieldset>
+            <div class="row" id="post_process_alternate">
+                <label>Force Post-Process Albums in Alternate Folder</label>
+			    <input type="text" value="" name="dir" id="dir" size="50" />
+			    <input type="button" class="btnOpenDialog" value="Submit" />
+            </div>
+        </fieldset>
+
+        <fieldset>
+            <div class="row" id="post_process_single">
+                <label>Post-Process Single Folder</label>
+			    <input type="text" value="" name="album_dir" id="album_dir" size="50" />
+			    <input type="button" class="btnOpenDialog" value="Submit" />
+            </div>
+        </fieldset>
+
 	</div>
 
 
@@ -167,9 +166,9 @@
 			<legend>Force Legacy</legend>
 				<p>Please note that these functions will take a significant amount of time to complete.</p>
 			<div class="links">
-				<a href="#" onclick="doAjaxCall('forceFullUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Comprehensive]</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceFullUpdate',$(this))" data-success="Update active artists successful" data-error="Error forcing update artists"><i class="fa fa-heart fa-fw"></i> Force Update Active Artists [Comprehensive]</a>
 				<BR>
-				<a href="#" onclick="doAjaxCall('forceScan',$(this))" data-success="Library scan successful" data-error="Error forcing library scan"><i class="fa fa-refresh fa-fw"></i> Force Re-scan Library [Comprehensive]</a>
+				<a href="javascript:void(0)" onclick="doAjaxCall('forceScan',$(this))" data-success="Library scan successful" data-error="Error forcing library scan"><i class="fa fa-refresh fa-fw"></i> Force Re-scan Library [Comprehensive]</a>
 				<BR>
 				<small>*Warning: If you choose [Force Re-scan Library], any manually ignored/matched artists/albums will be reset to "unmatched".</small>
 
@@ -177,8 +176,7 @@
 		</fieldset>
 		
 	</div>
-
-
+	<div id="dialog-confirm"></div>
 
 </div>
 </%def>
@@ -187,6 +185,55 @@
 		function addScanAction() {
 			$('#autoadd').append('<input type="hidden" name="scan" value=1 />');
 		};
+
+		function fnOpenNormalDialog(id) {
+			if (id === "post_process"){
+				var url = "forcePostProcess?"
+			}
+			if (id === "post_process_alternate"){
+				var dir = $('#dir').val();
+				var url = "forcePostProcess?dir=" + dir + "&"
+			}
+			if (id === "post_process_single"){
+				var dir = $('#album_dir').val();
+				var url = "forcePostProcess?album_dir=" + dir + "&"
+			}
+			var t = $('<a data-success="Post-Processor is being loaded" data-error="Error during Post-Processing">');
+		    $("#dialog-confirm").html("Do you want to keep the original folder(s) after post-processing? If you click no, the folders still might be kept depending on your global settings");
+
+		    // Define the Dialog and its properties.
+		    $("#dialog-confirm").dialog({
+		        resizable: false,
+		        modal: true,
+		        title: "Keep original folder(s)?",
+		        height: 170,
+		        width: 400,
+		        buttons: {
+		            "Yes": function () {
+		                $(this).dialog('close');
+		                doAjaxCall(url + "keep_original_folder=True", t);
+		            },
+		                "No": function () {
+		                $(this).dialog('close');
+		                doAjaxCall(url + "keep_original_folder=False", t);
+		            }
+		        }
+		    });
+		}
+
+		$('.btnOpenDialog').click(function(e){
+			e.preventDefault();
+			var parentId = $(this).closest('div').prop('id');
+			fnOpenNormalDialog(parentId);
+		});
+
+		function callback(value) {
+		    if (value) {
+		        alert("Confirmed");
+		    } else {
+		        alert("Rejected");
+		    }
+		}
 		
 		function initThisPage() {
 			$('#manage_albums').click(function() {

data/interfaces/default/managealbums.html

@@ -112,40 +112,43 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
+		function initThisPage() {
+			$('#album_table').dataTable({
+				"bDestroy": true,
+				"aoColumns": [
+	                null,
+					null,
+					null,
+					null,
+					null,
+					null,
+					{ "sType": "title-numeric"},
+					null,
+					null
+				],
+				"aoColumnDefs": [
+				          { 'bSortable': false, 'aTargets': [ 0 ] }
+				],
+				"oLanguage": {
+					"sLengthMenu":"Show _MENU_ albums per page",
+					"sEmptyTable": "No album information available",
+					"sInfo":"Showing _TOTAL_ albums",
+					"sInfoEmpty":"Showing 0 to 0 of 0 albums",
+					"sInfoFiltered":"(filtered from _MAX_ total albums)",
+					"sSearch": ""},
+				"bPaginate": false,
+				"aaSorting": [[5, 'desc']],
+				"fnDrawCallback": function (o) {
+	                // Jump to top of page
+	                $('html,body').scrollTop(0);
+	            }
 
-	function initThisPage() {
-		$('#album_table').dataTable({
-			"bDestroy": true,
-			"aoColumns": [
-                null,
-				null,
-				null,
-				null,
-				null,
-				null,
-				{ "sType": "title-numeric"},
-				null,
-				null
-			],
-			"aoColumnDefs": [
-			          { 'bSortable': false, 'aTargets': [ 0 ] }
-			],				
-			"oLanguage": {
-				"sLengthMenu":"Show _MENU_ albums per page",
-				"sEmptyTable": "No album information available",
-				"sInfo":"Showing _TOTAL_ albums",
-				"sInfoEmpty":"Showing 0 to 0 of 0 albums",
-				"sInfoFiltered":"(filtered from _MAX_ total albums)",
-				"sSearch": ""},
-			"bPaginate": false,
-			"aaSorting": [[5, 'desc']]
-
+			});
+			resetFilters("albums");
+		}
+		$(document).ready(function() {
+			initThisPage();
 		});
-		resetFilters("albums");
-	}
-	$(document).ready(function() {
-		initThisPage();
-	});
 	</script>
 </%def>
 

data/interfaces/default/manageartists.html

@@ -86,31 +86,31 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	function initThisPage() {
-		$('#artist_table').dataTable({	
-			"bDestroy":true,		
-			"aoColumns": [
-				null,
-				null,
-				{ "sType": "title-string"},
-				null,
-				{ "sType": "title-string"},
-				null
-			],
-			"oLanguage": {
-				"sSearch" : "",
-				"sEmptyTable": " "
-			},
-			"bStateSave": true,
-			"bPaginate": false
+		function initThisPage() {
+			$('#artist_table').dataTable({
+				"bDestroy": true,
+				"aoColumns": [
+					null,
+					null,
+					{ "sType": "title-string"},
+					null,
+					{ "sType": "title-string"},
+					null
+				],
+				"oLanguage": {
+					"sSearch" : "",
+					"sEmptyTable": " "
+				},
+				"bStateSave": true,
+				"bPaginate": false
+			});
+			resetFilters("artists");
+		}
+		$(document).ready(function() {
+			initThisPage();
+		});
+		$(window).load(function(){
+			initFancybox();
 		});
-		resetFilters("artists");
-	}
-	$(document).ready(function() {
-		initThisPage();
-	});
-	$(window).load(function(){
-		initFancybox();
-	});
 	</script>
 </%def>

data/interfaces/default/managemanual.html

@@ -17,7 +17,7 @@
 <%def name="body()">
 	<div class="table_wrapper">
 	<div id="manageheader" class="title">
-		<h1 class="clearfix"><i class="fa fa-music"></i> Manage Manually Changed Albums</h1>
+		<h1 class="clearfix"><i class="fa fa-music"></i> Manage Manually Matched Albums</h1>
 	</div>
 
 	<table class="display" id="artist_table">
@@ -37,30 +37,30 @@
 				old_album_clean = album['AlbumTitle'].replace('&','%26').replace('+', '%2B').replace("'","%27")
 			%>
 			<td id="artist">${album['ArtistName']}<BR>
-				<button id="restore_artist${count_albums}" onClick="restore_Artist(this.id)">(<-) Restore Artist</button>
-				<div id="restore_artist_dialog${count_albums}" title="Restore Artist" style="display:none">
+				<button id="reset_artist${count_albums}" onClick="reset_Artist(this.id)">(<-) Reset Artist</button>
+				<div id="reset_artist_dialog${count_albums}" title="Reset Artist" style="display:none">
 					<table>
-					<tr><td>Are you sure you want to restore Local Artist: ${album['ArtistName']} to unmatched?</td></tr>
+					<tr><td>Are you sure you want to reset Local Artist: ${album['ArtistName']} to unmatched?</td></tr>
 					<tr><td align="right"><BR>
 					%if album['AlbumStatus'] == "Ignored":
-						<button href="#" onclick="doAjaxCall('markManual?action=unignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Restore Artist</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully reset ${album['ArtistName']} to unmatched">Reset Artist</button>
 					%elif album['AlbumStatus'] == "Matched":
-						<button href="#" onclick="doAjaxCall('markManual?action=unmatchArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Restore Artist</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unmatchArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully restored ${album['ArtistName']} to unmatched">Reset Artist</button>
 					%endif
 					</td></tr>
 					</table>
 				</div>
 			</td>
 			<td id="album">${album['AlbumTitle']}<BR>
-				<button id="restore_album${count_albums}" onClick="restore_Album(this.id)">(<-) Restore Album</button>
-				<div id="restore_album_dialog${count_albums}" title="Restore Album" style="display:none">
+				<button id="reset_album${count_albums}" onClick="reset_Album(this.id)">(<-) Reset Album</button>
+				<div id="reset_album_dialog${count_albums}" title="Reset Album" style="display:none">
 					<table>
-					<tr><td>Are you sure you want to restore Local Album: ${album['AlbumTitle']} to unmatched?</td></tr>
+					<tr><td>Are you sure you want to reset Local Album: ${album['AlbumTitle']} to unmatched?</td></tr>
 					<tr><td align="right"><BR>
 					%if album['AlbumStatus'] == "Ignored":
-						<button href="#" onclick="doAjaxCall('markManual?action=unignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully restored ${album['AlbumTitle']} to unmatched">Restore Album</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
 					%elif album['AlbumStatus'] == "Matched":
-						<button href="#" onclick="doAjaxCall('markManual?action=unmatchAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully restored ${album['AlbumTitle']} to unmatched">Restore Album</button>
+						<button href="javascript:void(0)" onclick="doAjaxCall('markManual?action=unmatchAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully reset ${album['AlbumTitle']} to unmatched">Reset Album</button>
 					%endif
 					</td></tr>
 					</table>
@@ -85,35 +85,37 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	$(document).ready(function()
-	{
-		$('#artist_table').dataTable(
-			{
-				"bStateSave": true,
-				"bPaginate": true,
-				"oLanguage": {
-					"sSearch": "",
-	                "sLengthMenu":"Show _MENU_ albums per page",
-						"sInfo":"Showing _START_ to _END_ of _TOTAL_ albums",
-						"sInfoEmpty":"Showing 0 to 0 of 0 albums",
-						"sInfoFiltered":"(filtered from _MAX_ total albums)",
-						"sEmptyTable": " ",
-	                    },
-	            "sPaginationType": "full_numbers",
-	         });
+	$(document).ready(function() {
+		$('#artist_table').dataTable({
+			"bStateSave": true,
+			"bPaginate": true,
+			"oLanguage": {
+				"sSearch": "",
+                "sLengthMenu":"Show _MENU_ albums per page",
+					"sInfo":"Showing _START_ to _END_ of _TOTAL_ albums",
+					"sInfoEmpty":"Showing 0 to 0 of 0 albums",
+					"sInfoFiltered":"(filtered from _MAX_ total albums)",
+					"sEmptyTable": " ",
+                    },
+            "sPaginationType": "full_numbers",
+            "fnDrawCallback": function (o) {
+                // Jump to top of page
+                $('html,body').scrollTop(0);
+            }
+         });
 
-initActions();
+		initActions();
 	});
 
-	function restore_Artist(clicked_id) {
-		n=clicked_id.replace("restore_artist","");
-		$("#restore_artist_dialog"+n).dialog();
+	function reset_Artist(clicked_id) {
+		n=clicked_id.replace("reset_artist","");
+		$("#reset_artist_dialog"+n).dialog();
 		return false;
 	}
 
-	function restore_Album(clicked_id) {
-		n=clicked_id.replace("restore_album","");
-		$("#restore_album_dialog"+n).dialog();
+	function reset_Album(clicked_id) {
+		n=clicked_id.replace("reset_album","");
+		$("#reset_album_dialog"+n).dialog();
 		return false;
 	}
 

data/interfaces/default/managenew.html

@@ -6,7 +6,7 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a id="menu_link_scan" onclick="doAjaxCall('musicScan?path=${headphones.MUSIC_DIR}&redirect=manageNew',$(this))" data-success="Music library is getting scanned">Scan Music Library</a>
+			<a id="menu_link_scan" onclick="doAjaxCall('musicScan?path=${headphones.CONFIG.MUSIC_DIR}&redirect=manageNew',$(this))" data-success="Music library is getting scanned">Scan Music Library</a>
 		</div>		
 	</div>	
 	<a href="manage" class="back">&laquo; Back to manage overview</a>
@@ -53,18 +53,16 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	$(document).ready(function()
-	{
-		$('#artist_table').dataTable(
-			{
-				"aaSorting": [[1, 'asc']],
-				"bStateSave": false,
-				"bPaginate": false,
-				"oLanguage": {
-					"sSearch" : ""},
-				
-			});
-initActions();
+	$(document).ready(function() {
+		$('#artist_table').dataTable({
+			"aaSorting": [[1, 'asc']],
+			"bStateSave": false,
+			"bPaginate": false,
+			"oLanguage": {
+				"sSearch" : ""},
+		});
+
+		initActions();
 	});
 	</script>
 </%def>

data/interfaces/default/manageunmatched.html

@@ -52,7 +52,7 @@
 					<table>
 					<tr><td>Are you sure you want to ignore Local Artist: ${album['ArtistName']} from future matching?</td></tr>
 					<tr><td align="right"><BR>
-					<button href="#" onclick="doAjaxCall('markUnmatched?action=ignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully ignored ${album['ArtistName']} from future matching">Ignore Artist</button>
+					<button href="javascript:void(0)" onclick="doAjaxCall('markUnmatched?action=ignoreArtist&existing_artist=${old_artist_clean}', $(this), 'page');" data-success="Successfully ignored ${album['ArtistName']} from future matching">Ignore Artist</button>
 					</td></tr>
 					</table>
 				</div>
@@ -66,7 +66,7 @@
 						</select>
 						</td></tr>
 					<tr><td></td><td align="right"><BR>
-					<button href="#" onclick="artist_matcher(${count_albums}, '${old_artist_js}')">Match Artist</button>
+					<button href="javascript:void(0)" onclick="artist_matcher(${count_albums}, '${old_artist_js}')">Match Artist</button>
 					</td></tr>
 					</table>
 				</div>
@@ -78,7 +78,7 @@
 					<table>
 					<tr><td>Are you sure you want to ignore Local Album: ${album['AlbumTitle']} from future matching?</td></tr>
 					<tr><td align="right"><BR>
-					<button href="#" onclick="doAjaxCall('markUnmatched?action=ignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully ignored ${album['AlbumTitle']} from future matching">Ignore Album</button>
+					<button href="javascript:void(0)" onclick="doAjaxCall('markUnmatched?action=ignoreAlbum&existing_artist=${old_artist_clean}&existing_album=${old_album_clean}', $(this), 'page');" data-success="Successfully ignored ${album['AlbumTitle']} from future matching">Ignore Album</button>
 					</td></tr>
 					</table>
 				</div>
@@ -97,7 +97,7 @@
 					</select>
 					</td></tr>
 					<tr><td></td><td align="right"><BR>
-					<button href="#" onclick="album_matcher(${count_albums}, '${old_artist_js}', '${old_album_js}')">Match Album</button>
+					<button href="javascript:void(0)" onclick="album_matcher(${count_albums}, '${old_artist_js}', '${old_album_js}')">Match Album</button>
 					</td></tr>
 					</table>
 				</div>
@@ -118,22 +118,24 @@
 <%def name="javascriptIncludes()">
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 	<script>
-	$(document).ready(function()
-	{
-		$('#artist_table').dataTable(
-			{
-				"bStateSave": true,
-				"bPaginate": true,
-				"oLanguage": {
-					"sSearch": "",
-	                "sLengthMenu":"Show _MENU_ albums per page",
-						"sInfo":"Showing _START_ to _END_ of _TOTAL_ albums",
-						"sInfoEmpty":"Showing 0 to 0 of 0 albums",
-						"sInfoFiltered":"(filtered from _MAX_ total albums)",
-						"sEmptyTable": " ",
-	                    },
-	            "sPaginationType": "full_numbers",
-	         });
+	$(document).ready(function() {
+		$('#artist_table').dataTable({
+			"bStateSave": true,
+			"bPaginate": true,
+			"oLanguage": {
+				"sSearch": "",
+                "sLengthMenu":"Show _MENU_ albums per page",
+					"sInfo":"Showing _START_ to _END_ of _TOTAL_ albums",
+					"sInfoEmpty":"Showing 0 to 0 of 0 albums",
+					"sInfoFiltered":"(filtered from _MAX_ total albums)",
+					"sEmptyTable": " ",
+                    },
+            "sPaginationType": "full_numbers",
+            "fnDrawCallback": function (o) {
+                // Jump to top of page
+                $('html,body').scrollTop(0);
+            }
+         });
 
 initActions();
 	});

data/interfaces/default/searchresults.html

@@ -16,8 +16,13 @@
                     <th id="tracks">Tracks</th>
                     <th id="reldate">Date</th>
                     <th id="scoresmall">Score</th>
-                %else:
+                    <th id="mbrelid" style="display:none;"</th>
+                %elif type == 'artist':
                     <th id="artistname">Artist Name</th>
+                    <th id="score">Score</th>
+                %else:
+                    <th id="seriesname">Series Name</th>
+                    <th id="type">Type</th>
                     <th id="score">Score</th>
 				%endif
                 <th id="mb"></th>
@@ -38,9 +43,11 @@
                 %>
 				<tr class="grade${grade}">
 				%if type == 'album':
-					<td id="albumart" style=" text-align: center; vertical-align: middle;"><div id="artistImg"><img title="${result['albumid']}" class="albumArt" height="50" width="50" onerror="this.onerror=null;this.src='${caa_group_url}'"></div></td>
-                %else:
+					<td id="albumart" style=" text-align: center; vertical-align: middle;"><div id="artistImg"><img title="${result['albumid']}" class="albumArt" height="50" width="50" onerror="tryCCA(this, '${caa_group_url}')"></div></td>
+                %elif type == 'artist':
 					<td id="albumart"><div id="artistImg"><img title="${result['id']}" class="albumArt" height="50" width="50"></div></td>
+                %else:
+                    <td id="albumart"></td>
 				%endif
 				%if type == 'album':
 					<td id="albumname"><a href="addReleaseById?rid=${result['albumid']}&rgid=${result['rgid']}" title="${albuminfo}">${result['title']}</a></td>
@@ -50,9 +57,15 @@
                     <td id="reldate">${result['date']}</td>
                     <td id="score"><a href="${result['albumurl']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
                     <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['albumurl']}"><img src="interfaces/default/images/MusicBrainz_Album_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
-                %else:
+                    <td id="mbrelid" style="display:none;">${result['albumid']}</td>
+                %elif type == 'artist':
                     <td id="artistname"><a href="addArtist?artistid=${result['id']}" title="${result['uniquename']}">${result['uniquename']}</a></td>
                     <td id="score"><a href="${result['url']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
+                    <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['url']}"><img src="interfaces/default/images/MusicBrainz_Artist_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
+                %else:
+                    <td id="seriesname"><a href="addSeries?seriesid=${result['id']}" title="${result['uniquename']}">${result['uniquename']}</a></td>
+                    <td id="type">${result['type']}</td>
+                    <td id="score"><a href="${result['url']} "title="View on MusicBrainz"><div class="bar"><div class="score" style="width: ${result['score']}px">${result['score']}</div></div></a></td>
                     <td id="musicbrainz" style=" text-align: center; line-height: 0; vertical-align: middle;"><a href="${result['url']}"><img src="interfaces/default/images/MusicBrainz_Artist_Icon.png" title="View on MusicBrainz" height="20" width="20"></a></td>
 				%endif
 				</tr>
@@ -72,7 +85,13 @@
 	<script src="js/libs/jquery.dataTables.min.js"></script>
 
 	<script type="text/javascript">
-
+        function tryCCA(element, url) {
+            element.onerror = function() {
+                element.onerror = null;
+                element.src = "interfaces/default/images/no-cover-art.png";
+            };
+            element.src = url;
+        }
     	function getArt() {
     		$("table#searchresults_table tr td#albumart img").each(function(){
     			var id = $(this).attr('title');
@@ -85,7 +104,6 @@
     		});
     	}
     	function initThisPage() {
-            getArt();
     		$('#searchresults_table').dataTable({
 				"bDestroy": true,
 				"aoColumnDefs": [
@@ -100,11 +118,17 @@
 					"sSearch" : ""},
 				"iDisplayLength": 25,
 				"sPaginationType": "full_numbers",
-				"aaSorting": []
+				"aaSorting": [],
+                "fnDrawCallback": function (o) {
+                    // Jump to top of page
+                    $('html,body').scrollTop(0);
+                }
 			});
             $('#searchresults_table').on("draw.dt", function () {
+                getArt();
                 $("img").unveil();
             });
+            getArt();
 			resetFilters("album");
     	}
     	$(document).ready(function(){
@@ -112,4 +136,18 @@
             initThisPage();
     	});
 	</script>
+
+    <script type="text/javascript">
+        <%!
+            # Abuse JSON module for escaping JavaScript
+            import json
+        %>
+        $(document).ready(function() {
+            // Search parameter
+            $("#searchbar input[name=name]").val(${name | json.dumps});
+
+            // Album or artist
+            $("#searchbar select[name=type]").val(${type | json.dumps});
+        });
+    </script>
 </%def>

data/interfaces/default/upcoming.html

@@ -3,7 +3,7 @@
 <%def name="headerIncludes()">
 	<div id="subhead_container">
 		<div id="subhead_menu">
-			<a href="#" id="menu_link_scan" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search"></i> Force Check</a>
+			<a href="javascript:void(0)" id="menu_link_scan" onclick="doAjaxCall('forceSearch',$(this))" data-success="Checking for wanted albums successful" data-error="Error checking wanted albums"><i class="fa fa-search"></i> Force Check</a>
 		</div>
 	</div>
 </%def>
@@ -97,7 +97,7 @@
 				"oLanguage": {
 					"sEmptyTable": " "
 				},
-				"bDestroy":true,
+				"bDestroy": true,
 				"bFilter": false,
 				"bInfo": false,
 				"bPaginate": false

headphones/albumart.py

@@ -13,15 +13,18 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import request, db
+from headphones import request, db, logger
+
 
 def getAlbumArt(albumid):
     myDB = db.DBConnection()
-    asin = myDB.action('SELECT AlbumASIN from albums WHERE AlbumID=?', [albumid]).fetchone()[0]
+    asin = myDB.action(
+        'SELECT AlbumASIN from albums WHERE AlbumID=?', [albumid]).fetchone()[0]
 
     if asin:
         return 'http://ec1.images-amazon.com/images/P/%s.01.LZZZZZZZ.jpg' % asin
 
+
 def getCachedArt(albumid):
     from headphones import cache
 

headphones/albumswitcher.py

@@ -14,69 +14,86 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import headphones
-from headphones import db, logger
+from headphones import db, logger, cache
+
 
 def switch(AlbumID, ReleaseID):
-    '''
+    """
     Takes the contents from allalbums & alltracks (based on ReleaseID) and switches them into
     the albums & tracks table.
-    '''
+    """
+    logger.debug('Switching allalbums and alltracks')
     myDB = db.DBConnection()
-    oldalbumdata = myDB.action('SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
-    newalbumdata = myDB.action('SELECT * from allalbums WHERE ReleaseID=?', [ReleaseID]).fetchone()
-    newtrackdata = myDB.action('SELECT * from alltracks WHERE ReleaseID=?', [ReleaseID]).fetchall()
+    oldalbumdata = myDB.action(
+        'SELECT * from albums WHERE AlbumID=?', [AlbumID]).fetchone()
+    newalbumdata = myDB.action(
+        'SELECT * from allalbums WHERE ReleaseID=?', [ReleaseID]).fetchone()
+    newtrackdata = myDB.action(
+        'SELECT * from alltracks WHERE ReleaseID=?', [ReleaseID]).fetchall()
     myDB.action('DELETE from tracks WHERE AlbumID=?', [AlbumID])
 
-    controlValueDict = {"AlbumID":  AlbumID}
+    controlValueDict = {"AlbumID": AlbumID}
 
-    newValueDict = {"ArtistID":         newalbumdata['ArtistID'],
-                    "ArtistName":       newalbumdata['ArtistName'],
-                    "AlbumTitle":       newalbumdata['AlbumTitle'],
-                    "ReleaseID":        newalbumdata['ReleaseID'],
-                    "AlbumASIN":        newalbumdata['AlbumASIN'],
-                    "ReleaseDate":      newalbumdata['ReleaseDate'],
-                    "Type":             newalbumdata['Type'],
-                    "ReleaseCountry":   newalbumdata['ReleaseCountry'],
-                    "ReleaseFormat":    newalbumdata['ReleaseFormat']
-                }
+    newValueDict = {"ArtistID": newalbumdata['ArtistID'],
+                    "ArtistName": newalbumdata['ArtistName'],
+                    "AlbumTitle": newalbumdata['AlbumTitle'],
+                    "ReleaseID": newalbumdata['ReleaseID'],
+                    "AlbumASIN": newalbumdata['AlbumASIN'],
+                    "ReleaseDate": newalbumdata['ReleaseDate'],
+                    "Type": newalbumdata['Type'],
+                    "ReleaseCountry": newalbumdata['ReleaseCountry'],
+                    "ReleaseFormat": newalbumdata['ReleaseFormat']
+                    }
 
     myDB.upsert("albums", newValueDict, controlValueDict)
 
+    # Update cache
+    c = cache.Cache()
+    c.remove_from_cache(AlbumID=AlbumID)
+    c.get_artwork_from_cache(AlbumID=AlbumID)
+
     for track in newtrackdata:
+        controlValueDict = {"TrackID": track['TrackID'],
+                            "AlbumID": AlbumID}
 
-        controlValueDict = {"TrackID":  track['TrackID'],
-                            "AlbumID":  AlbumID}
-
-        newValueDict = {"ArtistID":     track['ArtistID'],
-                    "ArtistName":       track['ArtistName'],
-                    "AlbumTitle":       track['AlbumTitle'],
-                    "AlbumASIN":        track['AlbumASIN'],
-                    "ReleaseID":        track['ReleaseID'],
-                    "TrackTitle":       track['TrackTitle'],
-                    "TrackDuration":    track['TrackDuration'],
-                    "TrackNumber":      track['TrackNumber'],
-                    "CleanName":        track['CleanName'],
-                    "Location":         track['Location'],
-                    "Format":           track['Format'],
-                    "BitRate":          track['BitRate']
-                    }
+        newValueDict = {"ArtistID": track['ArtistID'],
+                        "ArtistName": track['ArtistName'],
+                        "AlbumTitle": track['AlbumTitle'],
+                        "AlbumASIN": track['AlbumASIN'],
+                        "ReleaseID": track['ReleaseID'],
+                        "TrackTitle": track['TrackTitle'],
+                        "TrackDuration": track['TrackDuration'],
+                        "TrackNumber": track['TrackNumber'],
+                        "CleanName": track['CleanName'],
+                        "Location": track['Location'],
+                        "Format": track['Format'],
+                        "BitRate": track['BitRate']
+                        }
 
         myDB.upsert("tracks", newValueDict, controlValueDict)
 
-    # Mark albums as downloaded if they have at least 80% (by default, configurable) of the album
+    # Mark albums as downloaded if they have at least 80% (by default,
+    # configurable) of the album
     total_track_count = len(newtrackdata)
-    have_track_count = len(myDB.select('SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [AlbumID]))
+    have_track_count = len(myDB.select(
+        'SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [AlbumID]))
 
-    if oldalbumdata['Status'] == 'Skipped' and ((have_track_count/float(total_track_count)) >= (headphones.ALBUM_COMPLETION_PCT/100.0)):
-        myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', AlbumID])
+    if oldalbumdata['Status'] == 'Skipped' and ((have_track_count / float(total_track_count)) >= (
+        headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
+        myDB.action(
+            'UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', AlbumID])
 
     # Update have track counts on index
-    totaltracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")', [newalbumdata['ArtistID']]))
-    havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [newalbumdata['ArtistID']]))
+    totaltracks = len(myDB.select(
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")',
+        [newalbumdata['ArtistID']]))
+    havetracks = len(myDB.select(
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL',
+        [newalbumdata['ArtistID']]))
 
-    controlValueDict = {"ArtistID":     newalbumdata['ArtistID']}
+    controlValueDict = {"ArtistID": newalbumdata['ArtistID']}
 
-    newValueDict = {    "TotalTracks":      totaltracks,
-                        "HaveTracks":       havetracks}
+    newValueDict = {"TotalTracks": totaltracks,
+                    "HaveTracks": havetracks}
 
     myDB.upsert("artists", newValueDict, controlValueDict)

headphones/api.py

@@ -13,22 +13,25 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import db, mb, importer, searcher, cache, postprocessor, versioncheck, logger
-
-from xml.dom.minidom import Document
-
-import headphones
-import copy
 import json
 
-cmd_list = [ 'getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSimilar', 'getHistory', 'getLogs',
-            'findArtist', 'findAlbum', 'addArtist', 'delArtist', 'pauseArtist', 'resumeArtist', 'refreshArtist',
-            'addAlbum', 'queueAlbum', 'unqueueAlbum', 'forceSearch', 'forceProcess', 'getVersion', 'checkGithub',
-            'shutdown', 'restart', 'update', 'getArtistArt', 'getAlbumArt', 'getArtistInfo', 'getAlbumInfo',
-            'getArtistThumb', 'getAlbumThumb', 'choose_specific_download', 'download_specific_release']
+from headphones import db, mb, updater, importer, searcher, cache, postprocessor, versioncheck, \
+    logger
+import headphones
+
+cmd_list = ['getIndex', 'getArtist', 'getAlbum', 'getUpcoming', 'getWanted', 'getSnatched',
+            'getSimilar', 'getHistory', 'getLogs',
+            'findArtist', 'findAlbum', 'addArtist', 'delArtist', 'pauseArtist', 'resumeArtist',
+            'refreshArtist',
+            'addAlbum', 'queueAlbum', 'unqueueAlbum', 'forceSearch', 'forceProcess',
+            'forceActiveArtistsUpdate',
+            'getVersion', 'checkGithub', 'shutdown', 'restart', 'update', 'getArtistArt',
+            'getAlbumArt',
+            'getArtistInfo', 'getAlbumInfo', 'getArtistThumb', 'getAlbumThumb', 'clearLogs',
+            'choose_specific_download', 'download_specific_release']
+
 
 class Api(object):
-
     def __init__(self):
 
         self.apikey = None
@@ -41,16 +44,15 @@ class Api(object):
 
         self.callback = None
 
+    def checkParams(self, *args, **kwargs):
 
-    def checkParams(self,*args,**kwargs):
-
-        if not headphones.API_ENABLED:
+        if not headphones.CONFIG.API_ENABLED:
             self.data = 'API not enabled'
             return
-        if not headphones.API_KEY:
+        if not headphones.CONFIG.API_KEY:
             self.data = 'API key not generated'
             return
-        if len(headphones.API_KEY) != 32:
+        if len(headphones.CONFIG.API_KEY) != 32:
             self.data = 'API key not generated correctly'
             return
 
@@ -58,7 +60,7 @@ class Api(object):
             self.data = 'Missing api key'
             return
 
-        if kwargs['apikey'] != headphones.API_KEY:
+        if kwargs['apikey'] != headphones.CONFIG.API_KEY:
             self.data = 'Incorrect API key'
             return
         else:
@@ -82,9 +84,9 @@ class Api(object):
         if self.data == 'OK':
             logger.info('Recieved API command: %s', self.cmd)
             methodToCall = getattr(self, "_" + self.cmd)
-            result = methodToCall(**self.kwargs)
+            methodToCall(**self.kwargs)
             if 'callback' not in self.kwargs:
-                if type(self.data) == type(''):
+                if isinstance(self.data, basestring):
                     return self.data
                 else:
                     return json.dumps(self.data)
@@ -96,7 +98,7 @@ class Api(object):
         else:
             return self.data
 
-    def _dic_from_query(self,query):
+    def _dic_from_query(self, query):
 
         myDB = db.DBConnection()
         rows = myDB.select(query)
@@ -111,7 +113,8 @@ class Api(object):
 
     def _getIndex(self, **kwargs):
 
-        self.data = self._dic_from_query('SELECT * from artists order by ArtistSortName COLLATE NOCASE')
+        self.data = self._dic_from_query(
+            'SELECT * from artists order by ArtistSortName COLLATE NOCASE')
         return
 
     def _getArtist(self, **kwargs):
@@ -122,11 +125,15 @@ class Api(object):
         else:
             self.id = kwargs['id']
 
-        artist = self._dic_from_query('SELECT * from artists WHERE ArtistID="' + self.id + '"')
-        albums = self._dic_from_query('SELECT * from albums WHERE ArtistID="' + self.id + '" order by ReleaseDate DESC')
-        description = self._dic_from_query('SELECT * from descriptions WHERE ArtistID="' + self.id + '"')
+        artist = self._dic_from_query(
+            'SELECT * from artists WHERE ArtistID="' + self.id + '"')
+        albums = self._dic_from_query(
+            'SELECT * from albums WHERE ArtistID="' + self.id + '" order by ReleaseDate DESC')
+        description = self._dic_from_query(
+            'SELECT * from descriptions WHERE ArtistID="' + self.id + '"')
 
-        self.data = { 'artist': artist, 'albums': albums, 'description' : description }
+        self.data = {
+            'artist': artist, 'albums': albums, 'description': description}
         return
 
     def _getAlbum(self, **kwargs):
@@ -137,23 +144,35 @@ class Api(object):
         else:
             self.id = kwargs['id']
 
-        album = self._dic_from_query('SELECT * from albums WHERE AlbumID="' + self.id + '"')
-        tracks = self._dic_from_query('SELECT * from tracks WHERE AlbumID="' + self.id + '"')
-        description = self._dic_from_query('SELECT * from descriptions WHERE ReleaseGroupID="' + self.id + '"')
+        album = self._dic_from_query(
+            'SELECT * from albums WHERE AlbumID="' + self.id + '"')
+        tracks = self._dic_from_query(
+            'SELECT * from tracks WHERE AlbumID="' + self.id + '"')
+        description = self._dic_from_query(
+            'SELECT * from descriptions WHERE ReleaseGroupID="' + self.id + '"')
 
-        self.data = { 'album' : album, 'tracks' : tracks, 'description' : description }
+        self.data = {
+            'album': album, 'tracks': tracks, 'description': description}
         return
 
     def _getHistory(self, **kwargs):
-        self.data = self._dic_from_query('SELECT * from snatched WHERE status NOT LIKE "Seed%" order by DateAdded DESC')
+        self.data = self._dic_from_query(
+            'SELECT * from snatched WHERE status NOT LIKE "Seed%" order by DateAdded DESC')
         return
 
     def _getUpcoming(self, **kwargs):
-        self.data = self._dic_from_query("SELECT * from albums WHERE ReleaseDate > date('now') order by ReleaseDate DESC")
+        self.data = self._dic_from_query(
+            "SELECT * from albums WHERE ReleaseDate > date('now') order by ReleaseDate DESC")
         return
 
     def _getWanted(self, **kwargs):
-        self.data = self._dic_from_query("SELECT * from albums WHERE Status='Wanted'")
+        self.data = self._dic_from_query(
+            "SELECT * from albums WHERE Status='Wanted'")
+        return
+
+    def _getSnatched(self, **kwargs):
+        self.data = self._dic_from_query(
+            "SELECT * from albums WHERE Status='Snatched'")
         return
 
     def _getSimilar(self, **kwargs):
@@ -161,7 +180,13 @@ class Api(object):
         return
 
     def _getLogs(self, **kwargs):
-        pass
+        self.data = headphones.LOG_LIST
+        return
+
+    def _clearLogs(self, **kwargs):
+        headphones.LOG_LIST = []
+        self.data = 'Cleared log'
+        return
 
     def _findArtist(self, **kwargs):
         if 'name' not in kwargs:
@@ -170,7 +195,7 @@ class Api(object):
         if 'limit' in kwargs:
             limit = kwargs['limit']
         else:
-            limit=50
+            limit = 50
 
         self.data = mb.findArtist(kwargs['name'], limit)
 
@@ -181,7 +206,7 @@ class Api(object):
         if 'limit' in kwargs:
             limit = kwargs['limit']
         else:
-            limit=50
+            limit = 50
 
         self.data = mb.findRelease(kwargs['name'], limit)
 
@@ -194,7 +219,7 @@ class Api(object):
 
         try:
             importer.addArtisttoDB(self.id)
-        except Exception, e:
+        except Exception as e:
             self.data = e
 
         return
@@ -244,7 +269,7 @@ class Api(object):
 
         try:
             importer.addArtisttoDB(self.id)
-        except Exception, e:
+        except Exception as e:
             self.data = e
 
         return
@@ -258,7 +283,7 @@ class Api(object):
 
         try:
             importer.addReleaseById(self.id)
-        except Exception, e:
+        except Exception as e:
             self.data = e
 
         return
@@ -307,18 +332,26 @@ class Api(object):
         searcher.searchforalbum()
 
     def _forceProcess(self, **kwargs):
-        self.dir = None
-        if 'dir' in kwargs:
+        if 'album_dir' in kwargs:
+            album_dir = kwargs['album_dir']
+            dir = None
+            postprocessor.forcePostProcess(self, dir, album_dir)
+        elif 'dir' in kwargs:
             self.dir = kwargs['dir']
-        postprocessor.forcePostProcess(self.dir)
+            postprocessor.forcePostProcess(self.dir)
+        else:
+            postprocessor.forcePostProcess()
+
+    def _forceActiveArtistsUpdate(self, **kwargs):
+        updater.dbUpdate()
 
     def _getVersion(self, **kwargs):
         self.data = {
-            'git_path' : headphones.GIT_PATH,
-            'install_type' : headphones.INSTALL_TYPE,
-            'current_version' : headphones.CURRENT_VERSION,
-            'latest_version' : headphones.LATEST_VERSION,
-            'commits_behind' : headphones.COMMITS_BEHIND,
+            'git_path': headphones.CONFIG.GIT_PATH,
+            'install_type': headphones.INSTALL_TYPE,
+            'current_version': headphones.CURRENT_VERSION,
+            'latest_version': headphones.LATEST_VERSION,
+            'commits_behind': headphones.COMMITS_BEHIND,
         }
 
     def _checkGithub(self, **kwargs):
@@ -402,18 +435,18 @@ class Api(object):
         else:
             self.id = kwargs['id']
 
-        results = searcher.searchforalbum(self.id, choose_specific_download=True)
+        results = searcher.searchforalbum(
+            self.id, choose_specific_download=True)
 
         results_as_dicts = []
 
         for result in results:
-
             result_dict = {
-                'title':result[0],
-                'size':result[1],
-                'url':result[2],
-                'provider':result[3],
-                'kind':result[4]
+                'title': result[0],
+                'size': result[1],
+                'url': result[2],
+                'provider': result[3],
+                'kind': result[4]
             }
             results_as_dicts.append(result_dict)
 
@@ -421,7 +454,7 @@ class Api(object):
 
     def _download_specific_release(self, **kwargs):
 
-        expected_kwargs =['id', 'title','size','url','provider','kind']
+        expected_kwargs = ['id', 'title', 'size', 'url', 'provider', 'kind']
 
         for kwarg in expected_kwargs:
             if kwarg not in kwargs:
@@ -438,20 +471,24 @@ class Api(object):
         for kwarg in expected_kwargs:
             del kwargs[kwarg]
 
-        # Handle situations where the torrent url contains arguments that are parsed
+        # Handle situations where the torrent url contains arguments that are
+        # parsed
         if kwargs:
-            import urllib, urllib2
-            url = urllib2.quote(url, safe=":?/=&") + '&' + urllib.urlencode(kwargs)
+            import urllib
+            import urllib2
+            url = urllib2.quote(
+                url, safe=":?/=&") + '&' + urllib.urlencode(kwargs)
 
         try:
-            result = [(title,int(size),url,provider,kind)]
+            result = [(title, int(size), url, provider, kind)]
         except ValueError:
-            result = [(title,float(size),url,provider,kind)]
+            result = [(title, float(size), url, provider, kind)]
 
         logger.info(u"Making sure we can download the chosen result")
         (data, bestqual) = searcher.preprocess(result)
 
         if data and bestqual:
-          myDB = db.DBConnection()
-          album = myDB.action('SELECT * from albums WHERE AlbumID=?', [id]).fetchone()
-          searcher.send_to_downloader(data, bestqual, album)
+            myDB = db.DBConnection()
+            album = myDB.action(
+                'SELECT * from albums WHERE AlbumID=?', [id]).fetchone()
+            searcher.send_to_downloader(data, bestqual, album)

headphones/cache.py

@@ -14,38 +14,39 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import glob
-import urllib
-import headphones
 
+import headphones
 from headphones import db, helpers, logger, lastfm, request
 
 LASTFM_API_KEY = "690e1ed3bc00bc91804cd8f7fe5ed6d4"
 
+
 class Cache(object):
     """
-    This class deals with getting, storing and serving up artwork (album
-    art, artist images, etc) and info/descriptions (album info, artist descrptions)
-    to and from the cache folder. This can be called from within a web interface,
-    for example, using the helper functions getInfo(id) and getArtwork(id), to utilize the cached
-    images rather than having to retrieve them every time the page is reloaded.
+    This class deals with getting, storing and serving up artwork (album art,
+    artist images, etc) and info/descriptions (album info, artist descrptions)
+    to and from the cache folder. This can be called from within a web
+    interface. For example, using the helper functions `getInfo(id)` and
+    `getArtwork(id)`, to utilize the cached images rather than having to
+    retrieve them every time the page is reloaded.
 
-    So you can call cache.getArtwork(id) which will return an absolute path
-    to the image file on the local machine, or if the cache directory
-    doesn't exist, or can not be written to, it will return a url to the image.
+    You can call `getArtwork(id)` which will return an absolute path to the
+    image file on the local machine, or if the cache directory doesn't exist,
+    or can not be written to, it will return a url to the image.
 
-    Call cache.getInfo(id) to grab the artist/album info; will return the text description
+    Call `getInfo(id)` to grab the artist/album info. This will return the
+    text description.
 
-    The basic format for art in the cache is <musicbrainzid>.<date>.<ext>
-    and for info it is <musicbrainzid>.<date>.txt
+    The basic format for art in the cache is `<musicbrainzid>.<date>.<ext>`
+    and for info it is `<musicbrainzid>.<date>.txt`
     """
 
-    path_to_art_cache = os.path.join(headphones.CACHE_DIR, 'artwork')
+    path_to_art_cache = os.path.join(headphones.CONFIG.CACHE_DIR, 'artwork')
 
     def __init__(self):
         self.id = None
-        self.id_type = None # 'artist' or 'album' - set automatically depending on whether ArtistID or AlbumID is passed
-        self.query_type = None # 'artwork','thumb' or 'info' - set automatically
+        self.id_type = None  # 'artist' or 'album' - set automatically depending on whether ArtistID or AlbumID is passed
+        self.query_type = None  # 'artwork','thumb' or 'info' - set automatically
 
         self.artwork_files = []
         self.thumb_files = []
@@ -59,12 +60,12 @@ class Cache(object):
         self.info_summary = None
         self.info_content = None
 
-    def _findfilesstartingwith(self,pattern,folder):
+    def _findfilesstartingwith(self, pattern, folder):
         files = []
         if os.path.exists(folder):
             for fname in os.listdir(folder):
                 if fname.startswith(pattern):
-                    files.append(os.path.join(folder,fname))
+                    files.append(os.path.join(folder, fname))
         return files
 
     def _exists(self, type):
@@ -72,14 +73,14 @@ class Cache(object):
         self.thumb_files = []
 
         if type == 'artwork':
-            self.artwork_files = self._findfilesstartingwith(self.id,self.path_to_art_cache)
+            self.artwork_files = self._findfilesstartingwith(self.id, self.path_to_art_cache)
             if self.artwork_files:
                 return True
             else:
                 return False
 
         elif type == 'thumb':
-            self.thumb_files = self._findfilesstartingwith("T_" + self.id,self.path_to_art_cache)
+            self.thumb_files = self._findfilesstartingwith("T_" + self.id, self.path_to_art_cache)
             if self.thumb_files:
                 return True
             else:
@@ -88,11 +89,10 @@ class Cache(object):
     def _get_age(self, date):
         # There's probably a better way to do this
         split_date = date.split('-')
-        days_old = int(split_date[0])*365 + int(split_date[1])*30 + int(split_date[2])
+        days_old = int(split_date[0]) * 365 + int(split_date[1]) * 30 + int(split_date[2])
 
         return days_old
 
-
     def _is_current(self, filename=None, date=None):
 
         if filename:
@@ -182,27 +182,32 @@ class Cache(object):
         if ArtistID:
             self.id = ArtistID
             self.id_type = 'artist'
-            db_info = myDB.action('SELECT Summary, Content, LastUpdated FROM descriptions WHERE ArtistID=?', [self.id]).fetchone()
+            db_info = myDB.action(
+                'SELECT Summary, Content, LastUpdated FROM descriptions WHERE ArtistID=?',
+                [self.id]).fetchone()
         else:
             self.id = AlbumID
             self.id_type = 'album'
-            db_info = myDB.action('SELECT Summary, Content, LastUpdated FROM descriptions WHERE ReleaseGroupID=?', [self.id]).fetchone()
+            db_info = myDB.action(
+                'SELECT Summary, Content, LastUpdated FROM descriptions WHERE ReleaseGroupID=?',
+                [self.id]).fetchone()
 
-        if not db_info or not db_info['LastUpdated'] or not self._is_current(date=db_info['LastUpdated']):
+        if not db_info or not db_info['LastUpdated'] or not self._is_current(
+                date=db_info['LastUpdated']):
 
             self._update_cache()
-            info_dict = { 'Summary' : self.info_summary, 'Content' : self.info_content }
+            info_dict = {'Summary': self.info_summary, 'Content': self.info_content}
             return info_dict
 
         else:
-            info_dict = { 'Summary' : db_info['Summary'], 'Content' : db_info['Content'] }
+            info_dict = {'Summary': db_info['Summary'], 'Content': db_info['Content']}
             return info_dict
 
     def get_image_links(self, ArtistID=None, AlbumID=None):
-        '''
+        """
         Here we're just going to open up the last.fm url, grab the image links and return them
         Won't save any image urls, or save the artwork in the cache. Useful for search results, etc.
-        '''
+        """
         if ArtistID:
 
             self.id_type = 'artist'
@@ -240,15 +245,47 @@ class Cache(object):
             if not thumb_url:
                 logger.debug('No album thumbnail image found on last.fm')
 
-        return {'artwork' : image_url, 'thumbnail' : thumb_url }
+        return {'artwork': image_url, 'thumbnail': thumb_url}
+
+    def remove_from_cache(self, ArtistID=None, AlbumID=None):
+        """
+        Pass a musicbrainz id to this function (either ArtistID or AlbumID)
+        """
+
+        if ArtistID:
+            self.id = ArtistID
+            self.id_type = 'artist'
+        else:
+            self.id = AlbumID
+            self.id_type = 'album'
+
+        self.query_type = 'artwork'
+
+        if self._exists('artwork'):
+            for artwork_file in self.artwork_files:
+                try:
+                    os.remove(artwork_file)
+                except:
+                    logger.warn('Error deleting file from the cache: %s', artwork_file)
+
+        self.query_type = 'thumb'
+
+        if self._exists('thumb'):
+            for thumb_file in self.thumb_files:
+                try:
+                    os.remove(thumb_file)
+                except Exception:
+                    logger.warn('Error deleting file from the cache: %s', thumb_file)
 
     def _update_cache(self):
-        '''
+        """
         Since we call the same url for both info and artwork, we'll update both at the same time
-        '''
+        """
+
         myDB = db.DBConnection()
 
         # Since lastfm uses release ids rather than release group ids for albums, we have to do a artist + album search for albums
+        # Exception is when adding albums manually, then we should use release id
         if self.id_type == 'artist':
 
             data = lastfm.request_lastfm("artist.getinfo", mbid=self.id, api_key=LASTFM_API_KEY)
@@ -277,9 +314,19 @@ class Cache(object):
                 logger.debug('No artist thumbnail image found')
 
         else:
-
-            dbartist = myDB.action('SELECT ArtistName, AlbumTitle FROM albums WHERE AlbumID=?', [self.id]).fetchone()
-            data = lastfm.request_lastfm("album.getinfo", artist=dbartist['ArtistName'], album=dbartist['AlbumTitle'], api_key=LASTFM_API_KEY)
+            dbalbum = myDB.action(
+                'SELECT ArtistName, AlbumTitle, ReleaseID FROM albums WHERE AlbumID=?',
+                [self.id]).fetchone()
+            if dbalbum['ReleaseID'] != self.id:
+                data = lastfm.request_lastfm("album.getinfo", mbid=dbalbum['ReleaseID'],
+                                             api_key=LASTFM_API_KEY)
+                if not data:
+                    data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'],
+                                                 album=dbalbum['AlbumTitle'],
+                                                 api_key=LASTFM_API_KEY)
+            else:
+                data = lastfm.request_lastfm("album.getinfo", artist=dbalbum['ArtistName'],
+                                             album=dbalbum['AlbumTitle'], api_key=LASTFM_API_KEY)
 
             if not data:
                 return
@@ -305,22 +352,24 @@ class Cache(object):
             if not thumb_url:
                 logger.debug('No album thumbnail image found')
 
-        #Save the content & summary to the database no matter what if we've opened up the url
+        # Save the content & summary to the database no matter what if we've
+        # opened up the url
         if self.id_type == 'artist':
-            controlValueDict = {"ArtistID":     self.id}
+            controlValueDict = {"ArtistID": self.id}
         else:
-            controlValueDict = {"ReleaseGroupID":     self.id}
+            controlValueDict = {"ReleaseGroupID": self.id}
 
-        newValueDict = {"Summary":       self.info_summary,
-                        "Content":       self.info_content,
-                        "LastUpdated":   helpers.today()}
+        newValueDict = {"Summary": self.info_summary,
+                        "Content": self.info_content,
+                        "LastUpdated": helpers.today()}
 
         myDB.upsert("descriptions", newValueDict, controlValueDict)
 
         # Save the image URL to the database
         if image_url:
             if self.id_type == 'artist':
-                myDB.action('UPDATE artists SET ArtworkURL=? WHERE ArtistID=?', [image_url, self.id])
+                myDB.action('UPDATE artists SET ArtworkURL=? WHERE ArtistID=?',
+                            [image_url, self.id])
             else:
                 myDB.action('UPDATE albums SET ArtworkURL=? WHERE AlbumID=?', [image_url, self.id])
 
@@ -331,7 +380,8 @@ class Cache(object):
             else:
                 myDB.action('UPDATE albums SET ThumbURL=? WHERE AlbumID=?', [thumb_url, self.id])
 
-        # Should we grab the artwork here if we're just grabbing thumbs or info?? Probably not since the files can be quite big
+        # Should we grab the artwork here if we're just grabbing thumbs or
+        # info? Probably not since the files can be quite big
         if image_url and self.query_type == 'artwork':
             artwork = request.request_content(image_url, timeout=20)
 
@@ -340,12 +390,14 @@ class Cache(object):
                 if not os.path.isdir(self.path_to_art_cache):
                     try:
                         os.makedirs(self.path_to_art_cache)
-                    except Exception, e:
+                        os.chmod(self.path_to_art_cache,
+                                 int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
+                    except OSError as e:
                         logger.error('Unable to create artwork cache dir. Error: %s', e)
                         self.artwork_errors = True
                         self.artwork_url = image_url
 
-                #Delete the old stuff
+                # Delete the old stuff
                 for artwork_file in self.artwork_files:
                     try:
                         os.remove(artwork_file)
@@ -354,17 +406,22 @@ class Cache(object):
 
                 ext = os.path.splitext(image_url)[1]
 
-                artwork_path = os.path.join(self.path_to_art_cache, self.id + '.' + helpers.today() + ext)
+                artwork_path = os.path.join(self.path_to_art_cache,
+                                            self.id + '.' + helpers.today() + ext)
                 try:
                     with open(artwork_path, 'wb') as f:
                         f.write(artwork)
-                except IOError as e:
+
+                    os.chmod(artwork_path, int(headphones.CONFIG.FILE_PERMISSIONS, 8))
+                except (OSError, IOError) as e:
                     logger.error('Unable to write to the cache dir: %s', e)
                     self.artwork_errors = True
                     self.artwork_url = image_url
 
-        # Grab the thumbnail as well if we're getting the full artwork (as long as it's missing/outdated
-        if thumb_url and self.query_type in ['thumb','artwork'] and not (self.thumb_files and self._is_current(self.thumb_files[0])):
+        # Grab the thumbnail as well if we're getting the full artwork (as long
+        # as it's missing/outdated.
+        if thumb_url and self.query_type in ['thumb', 'artwork'] and not (
+            self.thumb_files and self._is_current(self.thumb_files[0])):
             artwork = request.request_content(thumb_url, timeout=20)
 
             if artwork:
@@ -372,31 +429,36 @@ class Cache(object):
                 if not os.path.isdir(self.path_to_art_cache):
                     try:
                         os.makedirs(self.path_to_art_cache)
-                    except Exception as e:
+                        os.chmod(self.path_to_art_cache,
+                                 int(headphones.CONFIG.FOLDER_PERMISSIONS, 8))
+                    except OSError as e:
                         logger.error('Unable to create artwork cache dir. Error: %s' + e)
                         self.thumb_errors = True
                         self.thumb_url = thumb_url
 
-                #Delete the old stuff
+                # Delete the old stuff
                 for thumb_file in self.thumb_files:
                     try:
                         os.remove(thumb_file)
-                    except Exception as e:
+                    except OSError as e:
                         logger.error('Error deleting file from the cache: %s', thumb_file)
 
                 ext = os.path.splitext(image_url)[1]
 
-                thumb_path = os.path.join(self.path_to_art_cache, 'T_' + self.id + '.' + helpers.today() + ext)
+                thumb_path = os.path.join(self.path_to_art_cache,
+                                          'T_' + self.id + '.' + helpers.today() + ext)
                 try:
                     with open(thumb_path, 'wb') as f:
                         f.write(artwork)
-                except IOError as e:
+
+                    os.chmod(thumb_path, int(headphones.CONFIG.FILE_PERMISSIONS, 8))
+                except (OSError, IOError) as e:
                     logger.error('Unable to write to the cache dir: %s', e)
                     self.thumb_errors = True
                     self.thumb_url = image_url
 
-def getArtwork(ArtistID=None, AlbumID=None):
 
+def getArtwork(ArtistID=None, AlbumID=None):
     c = Cache()
     artwork_path = c.get_artwork_from_cache(ArtistID, AlbumID)
 
@@ -409,8 +471,8 @@ def getArtwork(ArtistID=None, AlbumID=None):
         artwork_file = os.path.basename(artwork_path)
         return "cache/artwork/" + artwork_file
 
-def getThumb(ArtistID=None, AlbumID=None):
 
+def getThumb(ArtistID=None, AlbumID=None):
     c = Cache()
     artwork_path = c.get_thumb_from_cache(ArtistID, AlbumID)
 
@@ -423,16 +485,15 @@ def getThumb(ArtistID=None, AlbumID=None):
         thumbnail_file = os.path.basename(artwork_path)
         return "cache/artwork/" + thumbnail_file
 
+
 def getInfo(ArtistID=None, AlbumID=None):
-
     c = Cache()
-
     info_dict = c.get_info_from_cache(ArtistID, AlbumID)
 
     return info_dict
 
-def getImageLinks(ArtistID=None, AlbumID=None):
 
+def getImageLinks(ArtistID=None, AlbumID=None):
     c = Cache()
     image_links = c.get_image_links(ArtistID, AlbumID)
 

headphones/classes.py

@@ -13,35 +13,36 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-#########################################
-## Stolen from Sick-Beard's classes.py ##
-#########################################
+#######################################
+# Stolen from Sick-Beard's classes.py #
+#######################################
 
-import headphones
 
 import urllib
-import datetime
 
 from common import USER_AGENT
 
+
 class HeadphonesURLopener(urllib.FancyURLopener):
     version = USER_AGENT
 
+
 class AuthURLOpener(HeadphonesURLopener):
     """
     URLOpener class that supports http auth without needing interactive password entry.
     If the provided username/password don't work it simply fails.
-    
+
     user: username to use for HTTP auth
     pw: password to use for HTTP auth
     """
+
     def __init__(self, user, pw):
         self.username = user
         self.password = pw
 
         # remember if we've tried the username/password before
         self.numTries = 0
-        
+
         # call the base class
         urllib.FancyURLopener.__init__(self)
 
@@ -55,7 +56,7 @@ class AuthURLOpener(HeadphonesURLopener):
         if self.numTries == 0:
             self.numTries = 1
             return (self.username, self.password)
-        
+
         # if we've tried before then return blank which cancels the request
         else:
             return ('', '')
@@ -65,6 +66,7 @@ class AuthURLOpener(HeadphonesURLopener):
         self.numTries = 0
         return HeadphonesURLopener.open(self, url)
 
+
 class SearchResult:
     """
     Represents a search result from an indexer.
@@ -87,7 +89,7 @@ class SearchResult:
 
     def __str__(self):
 
-        if self.provider == None:
+        if self.provider is None:
             return "Invalid provider, unable to print self"
 
         myString = self.provider.name + " @ " + self.url + "\n"
@@ -96,24 +98,28 @@ class SearchResult:
             myString += "  " + extra + "\n"
         return myString
 
+
 class NZBSearchResult(SearchResult):
     """
     Regular NZB result with an URL to the NZB
     """
     resultType = "nzb"
 
+
 class NZBDataSearchResult(SearchResult):
     """
     NZB result where the actual NZB XML data is stored in the extraInfo
     """
     resultType = "nzbdata"
 
+
 class TorrentSearchResult(SearchResult):
     """
     Torrent result with an URL to the torrent
     """
     resultType = "torrent"
 
+
 class Proper:
     def __init__(self, name, url, date):
         self.name = name
@@ -127,4 +133,5 @@ class Proper:
         self.episode = -1
 
     def __str__(self):
-        return str(self.date)+" "+self.name+" "+str(self.season)+"x"+str(self.episode)+" of "+str(self.tvdbid)
+        return str(self.date) + " " + self.name + " " + str(self.season) + "x" + str(
+            self.episode) + " of " + str(self.tvdbid)

headphones/common.py

@@ -18,14 +18,18 @@ Created on Aug 1, 2011
 
 @author: Michael
 '''
-import platform, operator, os, re
+import platform
+import operator
 
+import os
+import re
 from headphones import version
 
-#Identify Our Application
-USER_AGENT = 'Headphones/-'+version.HEADPHONES_VERSION+' ('+platform.system()+' '+platform.release()+')'
 
-### Notification Types
+# Identify Our Application
+USER_AGENT = 'Headphones/-' + version.HEADPHONES_VERSION + ' (' + platform.system() + ' ' + platform.release() + ')'
+
+# Notification Types
 NOTIFY_SNATCH = 1
 NOTIFY_DOWNLOAD = 2
 
@@ -33,28 +37,28 @@ notifyStrings = {}
 notifyStrings[NOTIFY_SNATCH] = "Started Download"
 notifyStrings[NOTIFY_DOWNLOAD] = "Download Finished"
 
-### Release statuses
-UNKNOWN = -1 # should never happen
-UNAIRED = 1 # releases that haven't dropped yet
-SNATCHED = 2 # qualified with quality
-WANTED = 3 # releases we don't have but want to get
-DOWNLOADED = 4 # qualified with quality
-SKIPPED = 5 # releases we don't want
-ARCHIVED = 6 # releases that you don't have locally (counts toward download completion stats)
-IGNORED = 7 # releases that you don't want included in your download stats
-SNATCHED_PROPER = 9 # qualified with quality
+# Release statuses
+UNKNOWN = -1  # should never happen
+UNAIRED = 1  # releases that haven't dropped yet
+SNATCHED = 2  # qualified with quality
+WANTED = 3  # releases we don't have but want to get
+DOWNLOADED = 4  # qualified with quality
+SKIPPED = 5  # releases we don't want
+ARCHIVED = 6  # releases that you don't have locally (counts toward download completion stats)
+IGNORED = 7  # releases that you don't want included in your download stats
+SNATCHED_PROPER = 9  # qualified with quality
+
 
 class Quality:
-
     NONE = 0
-    B192 = 1<<1     # 2
-    VBR =  1<<2     # 4
-    B256 = 1<<3     # 8
-    B320 = 1<<4     #16
-    FLAC = 1<<5     #32
+    B192 = 1 << 1  # 2
+    VBR = 1 << 2  # 4
+    B256 = 1 << 3  # 8
+    B320 = 1 << 4  # 16
+    FLAC = 1 << 5  # 32
 
     # put these bits at the other end of the spectrum, far enough out that they shouldn't interfere
-    UNKNOWN = 1<<15
+    UNKNOWN = 1 << 15
 
     qualityStrings = {NONE: "N/A",
                       UNKNOWN: "Unknown",
@@ -71,7 +75,8 @@ class Quality:
     def _getStatusStrings(status):
         toReturn = {}
         for x in Quality.qualityStrings.keys():
-            toReturn[Quality.compositeStatus(status, x)] = Quality.statusPrefixes[status]+" ("+Quality.qualityStrings[x]+")"
+            toReturn[Quality.compositeStatus(status, x)] = Quality.statusPrefixes[status] + " (" + \
+                                                           Quality.qualityStrings[x] + ")"
         return toReturn
 
     @staticmethod
@@ -82,7 +87,7 @@ class Quality:
             anyQuality = reduce(operator.or_, anyQualities)
         if bestQualities:
             bestQuality = reduce(operator.or_, bestQualities)
-        return anyQuality | (bestQuality<<16)
+        return anyQuality | (bestQuality << 16)
 
     @staticmethod
     def splitQuality(quality):
@@ -91,7 +96,7 @@ class Quality:
         for curQual in Quality.qualityStrings.keys():
             if curQual & quality:
                 anyQualities.append(curQual)
-            if curQual<<16 & quality:
+            if curQual << 16 & quality:
                 bestQualities.append(curQual)
 
         return (anyQualities, bestQualities)
@@ -99,6 +104,9 @@ class Quality:
     @staticmethod
     def nameQuality(name):
 
+        def checkName(list, func):
+            return func([re.search(x, name, re.I) for x in list])
+
         name = os.path.basename(name)
 
         # if we have our exact text then assume we put it there
@@ -106,14 +114,12 @@ class Quality:
             if x == Quality.UNKNOWN:
                 continue
 
-            regex = '\W'+Quality.qualityStrings[x].replace(' ','\W')+'\W'
+            regex = '\W' + Quality.qualityStrings[x].replace(' ', '\W') + '\W'
             regex_match = re.search(regex, name, re.I)
             if regex_match:
                 return x
 
-        checkName = lambda list, func: func([re.search(x, name, re.I) for x in list])
-
-        #TODO: fix quality checking here
+        # TODO: fix quality checking here
         if checkName(["mp3", "192"], any) and not checkName(["flac"], all):
             return Quality.B192
         elif checkName(["mp3", "256"], any) and not checkName(["flac"], all):
@@ -127,7 +133,6 @@ class Quality:
 
     @staticmethod
     def assumeQuality(name):
-
         if name.lower().endswith(".mp3"):
             return Quality.MP3
         elif name.lower().endswith(".flac"):
@@ -147,8 +152,8 @@ class Quality:
     def splitCompositeStatus(status):
         """Returns a tuple containing (status, quality)"""
         for x in sorted(Quality.qualityStrings.keys(), reverse=True):
-            if status > x*100:
-                return (status-x*100, x)
+            if status > x * 100:
+                return (status - x * 100, x)
 
         return (Quality.NONE, status)
 
@@ -163,13 +168,16 @@ class Quality:
     SNATCHED = None
     SNATCHED_PROPER = None
 
+
 Quality.DOWNLOADED = [Quality.compositeStatus(DOWNLOADED, x) for x in Quality.qualityStrings.keys()]
 Quality.SNATCHED = [Quality.compositeStatus(SNATCHED, x) for x in Quality.qualityStrings.keys()]
-Quality.SNATCHED_PROPER = [Quality.compositeStatus(SNATCHED_PROPER, x) for x in Quality.qualityStrings.keys()]
+Quality.SNATCHED_PROPER = [Quality.compositeStatus(SNATCHED_PROPER, x) for x in
+                           Quality.qualityStrings.keys()]
 
 MP3 = Quality.combineQualities([Quality.B192, Quality.B256, Quality.B320, Quality.VBR], [])
 LOSSLESS = Quality.combineQualities([Quality.FLAC], [])
-ANY = Quality.combineQualities([Quality.B192, Quality.B256, Quality.B320, Quality.VBR, Quality.FLAC], [])
+ANY = Quality.combineQualities(
+    [Quality.B192, Quality.B256, Quality.B320, Quality.VBR, Quality.FLAC], [])
 
 qualityPresets = (MP3, LOSSLESS, ANY)
 qualityPresetStrings = {MP3: "MP3 (All bitrates 192+)",

headphones/config.py

@@ -0,0 +1,494 @@
+import itertools
+
+import os
+import re
+import headphones.logger
+from configobj import ConfigObj
+
+
+def bool_int(value):
+    """
+    Casts a config value into a 0 or 1
+    """
+    if isinstance(value, basestring):
+        if value.lower() in ('', '0', 'false', 'f', 'no', 'n', 'off'):
+            value = 0
+    return int(bool(value))
+
+
+_CONFIG_DEFINITIONS = {
+    'ADD_ALBUM_ART': (int, 'General', 0),
+    'ADVANCEDENCODER': (str, 'General', ''),
+    'ALBUM_ART_FORMAT': (str, 'General', 'folder'),
+    # This is used in importer.py to determine how complete an album needs to
+    # be - to be considered "downloaded". Percentage from 0-100
+    'ALBUM_COMPLETION_PCT': (int, 'Advanced', 80),
+    'API_ENABLED': (int, 'General', 0),
+    'API_KEY': (str, 'General', ''),
+    'AUTOWANT_ALL': (int, 'General', 0),
+    'AUTOWANT_MANUALLY_ADDED': (int, 'General', 1),
+    'AUTOWANT_UPCOMING': (int, 'General', 1),
+    'AUTO_ADD_ARTISTS': (int, 'General', 1),
+    'BITRATE': (int, 'General', 192),
+    'BLACKHOLE': (int, 'General', 0),
+    'BLACKHOLE_DIR': (str, 'General', ''),
+    'BOXCAR_ENABLED': (int, 'Boxcar', 0),
+    'BOXCAR_ONSNATCH': (int, 'Boxcar', 0),
+    'BOXCAR_TOKEN': (str, 'Boxcar', ''),
+    'CACHE_DIR': (str, 'General', ''),
+    'CACHE_SIZEMB': (int, 'Advanced', 32),
+    'CHECK_GITHUB': (int, 'General', 1),
+    'CHECK_GITHUB_INTERVAL': (int, 'General', 360),
+    'CHECK_GITHUB_ON_STARTUP': (int, 'General', 1),
+    'CLEANUP_FILES': (int, 'General', 0),
+    'CONFIG_VERSION': (str, 'General', '0'),
+    'CORRECT_METADATA': (int, 'General', 0),
+    'CUE_SPLIT': (int, 'General', 1),
+    'CUE_SPLIT_FLAC_PATH': (str, 'General', ''),
+    'CUE_SPLIT_SHNTOOL_PATH': (str, 'General', ''),
+    'CUSTOMAUTH': (int, 'General', 0),
+    'CUSTOMHOST': (str, 'General', 'localhost'),
+    'CUSTOMPASS': (str, 'General', ''),
+    'CUSTOMPORT': (int, 'General', 5000),
+    'CUSTOMSLEEP': (int, 'General', 1),
+    'CUSTOMUSER': (str, 'General', ''),
+    'DELETE_LOSSLESS_FILES': (int, 'General', 1),
+    'DESTINATION_DIR': (str, 'General', ''),
+    'DETECT_BITRATE': (int, 'General', 0),
+    'DO_NOT_PROCESS_UNMATCHED': (int, 'General', 0),
+    'DOWNLOAD_DIR': (str, 'General', ''),
+    'DOWNLOAD_SCAN_INTERVAL': (int, 'General', 5),
+    'DOWNLOAD_TORRENT_DIR': (str, 'General', ''),
+    'DO_NOT_OVERRIDE_GIT_BRANCH': (int, 'General', 0),
+    'EMAIL_ENABLED': (int, 'Email', 0),
+    'EMAIL_FROM': (str, 'Email', ''),
+    'EMAIL_TO': (str, 'Email', ''),
+    'EMAIL_SMTP_SERVER': (str, 'Email', ''),
+    'EMAIL_SMTP_USER': (str, 'Email', ''),
+    'EMAIL_SMTP_PASSWORD': (str, 'Email', ''),
+    'EMAIL_SMTP_PORT': (int, 'Email', 25),
+    'EMAIL_SSL': (int, 'Email', 0),
+    'EMAIL_TLS': (int, 'Email', 0),
+    'EMAIL_ONSNATCH': (int, 'Email', 0),
+    'EMBED_ALBUM_ART': (int, 'General', 0),
+    'EMBED_LYRICS': (int, 'General', 0),
+    'ENABLE_HTTPS': (int, 'General', 0),
+    'ENCODER': (str, 'General', 'ffmpeg'),
+    'ENCODERFOLDER': (str, 'General', ''),
+    'ENCODERLOSSLESS': (int, 'General', 1),
+    'ENCODEROUTPUTFORMAT': (str, 'General', 'mp3'),
+    'ENCODERQUALITY': (int, 'General', 2),
+    'ENCODERVBRCBR': (str, 'General', 'cbr'),
+    'ENCODER_MULTICORE': (int, 'General', 0),
+    'ENCODER_MULTICORE_COUNT': (int, 'General', 0),
+    'ENCODER_PATH': (str, 'General', ''),
+    'EXTRAS': (str, 'General', ''),
+    'EXTRA_NEWZNABS': (list, 'Newznab', ''),
+    'EXTRA_TORZNABS': (list, 'Torznab', ''),
+    'FILE_FORMAT': (str, 'General', 'Track Artist - Album [Year] - Title'),
+    'FILE_PERMISSIONS': (str, 'General', '0644'),
+    'FILE_PERMISSIONS_ENABLED': (bool_int, 'General', True),
+    'FILE_UNDERSCORES': (int, 'General', 0),
+    'FOLDER_FORMAT': (str, 'General', 'Artist/Album [Year]'),
+    'FOLDER_PERMISSIONS_ENABLED': (bool_int, 'General', True),
+    'FOLDER_PERMISSIONS': (str, 'General', '0755'),
+    'FREEZE_DB': (int, 'General', 0),
+    'GIT_BRANCH': (str, 'General', 'master'),
+    'GIT_PATH': (str, 'General', ''),
+    'GIT_USER': (str, 'General', 'rembo10'),
+    'GROWL_ENABLED': (int, 'Growl', 0),
+    'GROWL_HOST': (str, 'Growl', ''),
+    'GROWL_ONSNATCH': (int, 'Growl', 0),
+    'GROWL_PASSWORD': (str, 'Growl', ''),
+    'HEADPHONES_INDEXER': (bool_int, 'General', False),
+    'HPPASS': (str, 'General', ''),
+    'HPUSER': (str, 'General', ''),
+    'HTTPS_CERT': (str, 'General', ''),
+    'HTTPS_KEY': (str, 'General', ''),
+    'HTTP_HOST': (str, 'General', 'localhost'),
+    'HTTP_PASSWORD': (str, 'General', ''),
+    'HTTP_PORT': (int, 'General', 8181),
+    'HTTP_PROXY': (int, 'General', 0),
+    'HTTP_ROOT': (str, 'General', '/'),
+    'HTTP_USERNAME': (str, 'General', ''),
+    'IDTAG': (int, 'Beets', 0),
+    'IGNORE_CLEAN_RELEASES': (int, 'General', 0),
+    'IGNORED_WORDS': (str, 'General', ''),
+    'IGNORED_FOLDERS': (list, 'Advanced', []),
+    'IGNORED_FILES': (list, 'Advanced', []),
+    'INCLUDE_EXTRAS': (int, 'General', 0),
+    'INTERFACE': (str, 'General', 'default'),
+    'JOURNAL_MODE': (str, 'Advanced', 'wal'),
+    'KAT': (int, 'Kat', 0),
+    'KAT_PROXY_URL': (str, 'Kat', ''),
+    'KAT_RATIO': (str, 'Kat', ''),
+    'KEEP_NFO': (int, 'General', 0),
+    'KEEP_TORRENT_FILES': (int, 'General', 0),
+    'LASTFM_USERNAME': (str, 'General', ''),
+    'LAUNCH_BROWSER': (int, 'General', 1),
+    'LIBRARYSCAN': (int, 'General', 1),
+    'LIBRARYSCAN_INTERVAL': (int, 'General', 300),
+    'LMS_ENABLED': (int, 'LMS', 0),
+    'LMS_HOST': (str, 'LMS', ''),
+    'LOG_DIR': (str, 'General', ''),
+    'LOSSLESS_BITRATE_FROM': (int, 'General', 0),
+    'LOSSLESS_BITRATE_TO': (int, 'General', 0),
+    'LOSSLESS_DESTINATION_DIR': (str, 'General', ''),
+    'MB_IGNORE_AGE': (int, 'General', 365),
+    'MININOVA': (int, 'Mininova', 0),
+    'MININOVA_RATIO': (str, 'Mininova', ''),
+    'MIRROR': (str, 'General', 'musicbrainz.org'),
+    'MOVE_FILES': (int, 'General', 0),
+    'MPC_ENABLED': (bool_int, 'MPC', False),
+    'MUSIC_DIR': (str, 'General', ''),
+    'MUSIC_ENCODER': (int, 'General', 0),
+    'NEWZNAB': (int, 'Newznab', 0),
+    'NEWZNAB_APIKEY': (str, 'Newznab', ''),
+    'NEWZNAB_ENABLED': (int, 'Newznab', 1),
+    'NEWZNAB_HOST': (str, 'Newznab', ''),
+    'NMA_APIKEY': (str, 'NMA', ''),
+    'NMA_ENABLED': (int, 'NMA', 0),
+    'NMA_ONSNATCH': (int, 'NMA', 0),
+    'NMA_PRIORITY': (int, 'NMA', 0),
+    'NUMBEROFSEEDERS': (str, 'General', '10'),
+    'NZBGET_CATEGORY': (str, 'NZBget', ''),
+    'NZBGET_HOST': (str, 'NZBget', ''),
+    'NZBGET_PASSWORD': (str, 'NZBget', ''),
+    'NZBGET_PRIORITY': (int, 'NZBget', 0),
+    'NZBGET_USERNAME': (str, 'NZBget', 'nzbget'),
+    'NZBSORG': (int, 'NZBsorg', 0),
+    'NZBSORG_HASH': (str, 'NZBsorg', ''),
+    'NZBSORG_UID': (str, 'NZBsorg', ''),
+    'NZB_DOWNLOADER': (int, 'General', 0),
+    'OFFICIAL_RELEASES_ONLY': (int, 'General', 0),
+    'OMGWTFNZBS': (int, 'omgwtfnzbs', 0),
+    'OMGWTFNZBS_APIKEY': (str, 'omgwtfnzbs', ''),
+    'OMGWTFNZBS_UID': (str, 'omgwtfnzbs', ''),
+    'OPEN_MAGNET_LINKS': (int, 'General', 0),  # 0: Ignore, 1: Open, 2: Convert, 3: Embed (rtorrent)
+    'MAGNET_LINKS': (int, 'General', 0),
+    'OSX_NOTIFY_APP': (str, 'OSX_Notify', '/Applications/Headphones'),
+    'OSX_NOTIFY_ENABLED': (int, 'OSX_Notify', 0),
+    'OSX_NOTIFY_ONSNATCH': (int, 'OSX_Notify', 0),
+    'PIRATEBAY': (int, 'Piratebay', 0),
+    'PIRATEBAY_PROXY_URL': (str, 'Piratebay', ''),
+    'PIRATEBAY_RATIO': (str, 'Piratebay', ''),
+    'OLDPIRATEBAY': (int, 'Old Piratebay', 0),
+    'OLDPIRATEBAY_URL': (str, 'Old Piratebay', ''),
+    'OLDPIRATEBAY_RATIO': (str, 'Old Piratebay', ''),
+    'PLEX_CLIENT_HOST': (str, 'Plex', ''),
+    'PLEX_ENABLED': (int, 'Plex', 0),
+    'PLEX_NOTIFY': (int, 'Plex', 0),
+    'PLEX_PASSWORD': (str, 'Plex', ''),
+    'PLEX_SERVER_HOST': (str, 'Plex', ''),
+    'PLEX_UPDATE': (int, 'Plex', 0),
+    'PLEX_USERNAME': (str, 'Plex', ''),
+    'PLEX_TOKEN': (str, 'Plex', ''),
+    'PREFERRED_BITRATE': (str, 'General', ''),
+    'PREFERRED_BITRATE_ALLOW_LOSSLESS': (int, 'General', 0),
+    'PREFERRED_BITRATE_HIGH_BUFFER': (int, 'General', 0),
+    'PREFERRED_BITRATE_LOW_BUFFER': (int, 'General', 0),
+    'PREFERRED_QUALITY': (int, 'General', 0),
+    'PREFERRED_WORDS': (str, 'General', ''),
+    'PREFER_TORRENTS': (int, 'General', 0),
+    'PROWL_ENABLED': (int, 'Prowl', 0),
+    'PROWL_KEYS': (str, 'Prowl', ''),
+    'PROWL_ONSNATCH': (int, 'Prowl', 0),
+    'PROWL_PRIORITY': (int, 'Prowl', 0),
+    'PUSHALOT_APIKEY': (str, 'Pushalot', ''),
+    'PUSHALOT_ENABLED': (int, 'Pushalot', 0),
+    'PUSHALOT_ONSNATCH': (int, 'Pushalot', 0),
+    'PUSHBULLET_APIKEY': (str, 'PushBullet', ''),
+    'PUSHBULLET_DEVICEID': (str, 'PushBullet', ''),
+    'PUSHBULLET_ENABLED': (int, 'PushBullet', 0),
+    'PUSHBULLET_ONSNATCH': (int, 'PushBullet', 0),
+    'PUSHOVER_APITOKEN': (str, 'Pushover', ''),
+    'PUSHOVER_ENABLED': (int, 'Pushover', 0),
+    'PUSHOVER_KEYS': (str, 'Pushover', ''),
+    'PUSHOVER_ONSNATCH': (int, 'Pushover', 0),
+    'PUSHOVER_PRIORITY': (int, 'Pushover', 0),
+    'RENAME_FILES': (int, 'General', 0),
+    'RENAME_UNPROCESSED': (bool_int, 'General', 1),
+    'RENAME_FROZEN': (bool_int, 'General', 1),
+    'REPLACE_EXISTING_FOLDERS': (int, 'General', 0),
+    'KEEP_ORIGINAL_FOLDER': (int, 'General', 0),
+    'REQUIRED_WORDS': (str, 'General', ''),
+    'RUTRACKER': (int, 'Rutracker', 0),
+    'RUTRACKER_PASSWORD': (str, 'Rutracker', ''),
+    'RUTRACKER_RATIO': (str, 'Rutracker', ''),
+    'RUTRACKER_USER': (str, 'Rutracker', ''),
+    'SAB_APIKEY': (str, 'SABnzbd', ''),
+    'SAB_CATEGORY': (str, 'SABnzbd', ''),
+    'SAB_HOST': (str, 'SABnzbd', ''),
+    'SAB_PASSWORD': (str, 'SABnzbd', ''),
+    'SAB_USERNAME': (str, 'SABnzbd', ''),
+    'SAMPLINGFREQUENCY': (int, 'General', 44100),
+    'SEARCH_INTERVAL': (int, 'General', 1440),
+    'SONGKICK_APIKEY': (str, 'Songkick', 'nd1We7dFW2RqxPw8'),
+    'SONGKICK_ENABLED': (int, 'Songkick', 1),
+    'SONGKICK_FILTER_ENABLED': (int, 'Songkick', 0),
+    'SONGKICK_LOCATION': (str, 'Songkick', ''),
+    'STRIKE': (int, 'Strike', 0),
+    'STRIKE_RATIO': (str, 'Strike', ''),
+    'SUBSONIC_ENABLED': (int, 'Subsonic', 0),
+    'SUBSONIC_HOST': (str, 'Subsonic', ''),
+    'SUBSONIC_PASSWORD': (str, 'Subsonic', ''),
+    'SUBSONIC_USERNAME': (str, 'Subsonic', ''),
+    'SYNOINDEX_ENABLED': (int, 'Synoindex', 0),
+    'TORRENTBLACKHOLE_DIR': (str, 'General', ''),
+    'TORRENT_DOWNLOADER': (int, 'General', 0),
+    'TORRENT_REMOVAL_INTERVAL': (int, 'General', 720),
+    'TORZNAB': (int, 'Torznab', 0),
+    'TORZNAB_APIKEY': (str, 'Torznab', ''),
+    'TORZNAB_ENABLED': (int, 'Torznab', 1),
+    'TORZNAB_HOST': (str, 'Torznab', ''),
+    'TRANSMISSION_HOST': (str, 'Transmission', ''),
+    'TRANSMISSION_PASSWORD': (str, 'Transmission', ''),
+    'TRANSMISSION_USERNAME': (str, 'Transmission', ''),
+    'TWITTER_ENABLED': (int, 'Twitter', 0),
+    'TWITTER_ONSNATCH': (int, 'Twitter', 0),
+    'TWITTER_PASSWORD': (str, 'Twitter', ''),
+    'TWITTER_PREFIX': (str, 'Twitter', 'Headphones'),
+    'TWITTER_USERNAME': (str, 'Twitter', ''),
+    'UPDATE_DB_INTERVAL': (int, 'General', 24),
+    'USENET_RETENTION': (int, 'General', '1500'),
+    'UTORRENT_HOST': (str, 'uTorrent', ''),
+    'UTORRENT_LABEL': (str, 'uTorrent', ''),
+    'UTORRENT_PASSWORD': (str, 'uTorrent', ''),
+    'UTORRENT_USERNAME': (str, 'uTorrent', ''),
+    'VERIFY_SSL_CERT': (bool_int, 'Advanced', 1),
+    'WAIT_UNTIL_RELEASE_DATE': (int, 'General', 0),
+    'WAFFLES': (int, 'Waffles', 0),
+    'WAFFLES_PASSKEY': (str, 'Waffles', ''),
+    'WAFFLES_RATIO': (str, 'Waffles', ''),
+    'WAFFLES_UID': (str, 'Waffles', ''),
+    'WHATCD': (int, 'What.cd', 0),
+    'WHATCD_PASSWORD': (str, 'What.cd', ''),
+    'WHATCD_RATIO': (str, 'What.cd', ''),
+    'WHATCD_USERNAME': (str, 'What.cd', ''),
+    'XBMC_ENABLED': (int, 'XBMC', 0),
+    'XBMC_HOST': (str, 'XBMC', ''),
+    'XBMC_NOTIFY': (int, 'XBMC', 0),
+    'XBMC_PASSWORD': (str, 'XBMC', ''),
+    'XBMC_UPDATE': (int, 'XBMC', 0),
+    'XBMC_USERNAME': (str, 'XBMC', ''),
+    'XLDPROFILE': (str, 'General', '')
+}
+
+
+# pylint:disable=R0902
+# it might be nice to refactor for fewer instance variables
+class Config(object):
+    """ Wraps access to particular values in a config file """
+
+    def __init__(self, config_file):
+        """ Initialize the config with values from a file """
+        self._config_file = config_file
+        self._config = ConfigObj(self._config_file, encoding='utf-8')
+        for key in _CONFIG_DEFINITIONS.keys():
+            self.check_setting(key)
+        self.ENCODER_MULTICORE_COUNT = max(0, self.ENCODER_MULTICORE_COUNT)
+        self._upgrade()
+
+    def _define(self, name):
+        key = name.upper()
+        ini_key = name.lower()
+        definition = _CONFIG_DEFINITIONS[key]
+        if len(definition) == 3:
+            definition_type, section, default = definition
+        else:
+            definition_type, section, _, default = definition
+        return key, definition_type, section, ini_key, default
+
+    def check_section(self, section):
+        """ Check if INI section exists, if not create it """
+        if section not in self._config:
+            self._config[section] = {}
+            return True
+        else:
+            return False
+
+    def check_setting(self, key):
+        """ Cast any value in the config to the right type or use the default """
+        key, definition_type, section, ini_key, default = self._define(key)
+        self.check_section(section)
+        try:
+            my_val = definition_type(self._config[section][ini_key])
+        except Exception:
+            my_val = definition_type(default)
+            self._config[section][ini_key] = my_val
+        return my_val
+
+    def write(self):
+        """ Make a copy of the stored config and write it to the configured file """
+        new_config = ConfigObj(encoding="UTF-8")
+        new_config.filename = self._config_file
+
+        # first copy over everything from the old config, even if it is not
+        # correctly defined to keep from losing data
+        for key, subkeys in self._config.items():
+            if key not in new_config:
+                new_config[key] = {}
+            for subkey, value in subkeys.items():
+                new_config[key][subkey] = value
+
+        # next make sure that everything we expect to have defined is so
+        for key in _CONFIG_DEFINITIONS.keys():
+            key, definition_type, section, ini_key, default = self._define(key)
+            self.check_setting(key)
+            if section not in new_config:
+                new_config[section] = {}
+            new_config[section][ini_key] = self._config[section][ini_key]
+
+        # Write it to file
+        headphones.logger.info("Writing configuration to file")
+
+        try:
+            new_config.write()
+        except IOError as e:
+            headphones.logger.error("Error writing configuration file: %s", e)
+
+    def get_extra_newznabs(self):
+        """ Return the extra newznab tuples """
+        extra_newznabs = list(
+            itertools.izip(*[itertools.islice(self.EXTRA_NEWZNABS, i, None, 3)
+                             for i in range(3)])
+        )
+        return extra_newznabs
+
+    def clear_extra_newznabs(self):
+        """ Forget about the configured extra newznabs """
+        self.EXTRA_NEWZNABS = []
+
+    def add_extra_newznab(self, newznab):
+        """ Add a new extra newznab """
+        extra_newznabs = self.EXTRA_NEWZNABS
+        for item in newznab:
+            extra_newznabs.append(item)
+        self.EXTRA_NEWZNABS = extra_newznabs
+
+    def get_extra_torznabs(self):
+        """ Return the extra torznab tuples """
+        extra_torznabs = list(
+            itertools.izip(*[itertools.islice(self.EXTRA_TORZNABS, i, None, 3)
+                             for i in range(3)])
+        )
+        return extra_torznabs
+
+    def clear_extra_torznabs(self):
+        """ Forget about the configured extra torznabs """
+        self.EXTRA_TORZNABS = []
+
+    def add_extra_torznab(self, torznab):
+        """ Add a new extra torznab """
+        extra_torznabs = self.EXTRA_TORZNABS
+        for item in torznab:
+            extra_torznabs.append(item)
+        self.EXTRA_TORZNABS = extra_torznabs
+
+    def __getattr__(self, name):
+        """
+        Returns something from the ini unless it is a real property
+        of the configuration object or is not all caps.
+        """
+        if not re.match(r'[A-Z_]+$', name):
+            return super(Config, self).__getattr__(name)
+        else:
+            return self.check_setting(name)
+
+    def __setattr__(self, name, value):
+        """
+        Maps all-caps properties to ini values unless they exist on the
+        configuration object.
+        """
+        if not re.match(r'[A-Z_]+$', name):
+            super(Config, self).__setattr__(name, value)
+            return value
+        else:
+            key, definition_type, section, ini_key, default = self._define(name)
+            self._config[section][ini_key] = definition_type(value)
+            return self._config[section][ini_key]
+
+    def process_kwargs(self, kwargs):
+        """
+        Given a big bunch of key value pairs, apply them to the ini.
+        """
+        for name, value in kwargs.items():
+            key, definition_type, section, ini_key, default = self._define(name)
+            self._config[section][ini_key] = definition_type(value)
+
+    def _upgrade(self):
+        """ Update folder formats in the config & bump up config version """
+        if self.CONFIG_VERSION == '0':
+            from headphones.helpers import replace_all
+            file_values = {
+                'tracknumber': 'Track',
+                'title': 'Title',
+                'artist': 'Artist',
+                'album': 'Album',
+                'year': 'Year'
+            }
+            folder_values = {
+                'artist': 'Artist',
+                'album': 'Album',
+                'year': 'Year',
+                'releasetype': 'Type',
+                'first': 'First',
+                'lowerfirst': 'first'
+            }
+            self.FILE_FORMAT = replace_all(self.FILE_FORMAT, file_values)
+            self.FOLDER_FORMAT = replace_all(self.FOLDER_FORMAT, folder_values)
+
+            self.CONFIG_VERSION = '1'
+
+        if self.CONFIG_VERSION == '1':
+            from headphones.helpers import replace_all
+            file_values = {
+                'Track': '$Track',
+                'Title': '$Title',
+                'Artist': '$Artist',
+                'Album': '$Album',
+                'Year': '$Year',
+                'track': '$track',
+                'title': '$title',
+                'artist': '$artist',
+                'album': '$album',
+                'year': '$year'
+            }
+            folder_values = {
+                'Artist': '$Artist',
+                'Album': '$Album',
+                'Year': '$Year',
+                'Type': '$Type',
+                'First': '$First',
+                'artist': '$artist',
+                'album': '$album',
+                'year': '$year',
+                'type': '$type',
+                'first': '$first'
+            }
+            self.FILE_FORMAT = replace_all(self.FILE_FORMAT, file_values)
+            self.FOLDER_FORMAT = replace_all(self.FOLDER_FORMAT, folder_values)
+            self.CONFIG_VERSION = '2'
+
+        if self.CONFIG_VERSION == '2':
+            # Update the config to use direct path to the encoder rather than the encoder folder
+            if self.ENCODERFOLDER:
+                self.ENCODER_PATH = os.path.join(self.ENCODERFOLDER, self.ENCODER)
+            self.CONFIG_VERSION = '3'
+
+        if self.CONFIG_VERSION == '3':
+            # Update the BLACKHOLE option to the NZB_DOWNLOADER format
+            if self.BLACKHOLE:
+                self.NZB_DOWNLOADER = 2
+            self.CONFIG_VERSION = '4'
+
+        # Enable Headphones Indexer if they have a VIP account
+        if self.CONFIG_VERSION == '4':
+            if self.HPUSER and self.HPPASS:
+                self.HEADPHONES_INDEXER = True
+            self.CONFIG_VERSION = '5'
+
+        if self.CONFIG_VERSION == '5':
+            if self.OPEN_MAGNET_LINKS:
+                self.MAGNET_LINKS = 2
+            self.CONFIG_VERSION = '5'

headphones/cuesplit.py

@@ -0,0 +1,674 @@
+#  This file is part of Headphones.
+#
+#  Headphones is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  Headphones is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
+
+# Most of this lifted from here: https://github.com/SzieberthAdam/gneposis-cdgrab
+
+import sys
+import subprocess
+import copy
+import glob
+
+import os
+import re
+import headphones
+from headphones import logger
+from mutagen.flac import FLAC
+
+CUE_HEADER = {
+    'genre': '^REM GENRE (.+?)$',
+    'date': '^REM DATE (.+?)$',
+    'discid': '^REM DISCID (.+?)$',
+    'comment': '^REM COMMENT (.+?)$',
+    'catalog': '^CATALOG (.+?)$',
+    'artist': '^PERFORMER (.+?)$',
+    'title': '^TITLE (.+?)$',
+    'file': '^FILE (.+?) (WAVE|FLAC)$',
+    'accurateripid': '^REM ACCURATERIPID (.+?)$'
+}
+
+CUE_TRACK = 'TRACK (\d\d) AUDIO$'
+
+CUE_TRACK_INFO = {
+    'artist': 'PERFORMER (.+?)$',
+    'title': 'TITLE (.+?)$',
+    'isrc': 'ISRC (.+?)$',
+    'index': 'INDEX (\d\d) (.+?)$'
+}
+
+ALBUM_META_FILE_NAME = 'album.dat'
+SPLIT_FILE_NAME = 'split.dat'
+
+ALBUM_META_ALBUM_BY_CUE = ('artist', 'title', 'date', 'genre')
+
+HTOA_LENGTH_TRIGGER = 3
+
+WAVE_FILE_TYPE_BY_EXTENSION = {
+    '.wav': 'Waveform Audio',
+    '.wv': 'WavPack',
+    '.ape': "Monkey's Audio",
+    '.m4a': 'Apple Lossless',
+    '.flac': 'Free Lossless Audio Codec'
+}
+
+# SHNTOOL_COMPATIBLE = ("Free Lossless Audio Codec", "Waveform Audio", "Monkey's Audio")
+
+# TODO: Make this better!
+# this module-level variable is bad. :(
+CUE_META = None
+
+
+def check_splitter(command):
+    '''Check xld or shntool installed'''
+    try:
+        env = os.environ.copy()
+        if 'xld' in command:
+            env['PATH'] += os.pathsep + '/Applications'
+        elif headphones.CONFIG.CUE_SPLIT_FLAC_PATH:
+            command = os.path.join(headphones.CONFIG.CUE_SPLIT_SHNTOOL_PATH, 'shntool')
+        devnull = open(os.devnull)
+        subprocess.Popen([command], stdout=devnull, stderr=devnull, env=env).communicate()
+    except OSError as e:
+        if e.errno == os.errno.ENOENT:
+            return False
+    return True
+
+
+def split_baby(split_file, split_cmd):
+    '''Let's split baby'''
+    logger.info('Splitting %s...', split_file.decode(headphones.SYS_ENCODING, 'replace'))
+    logger.debug(subprocess.list2cmdline(split_cmd))
+
+    # Prevent Windows from opening a terminal window
+    startupinfo = None
+
+    if headphones.SYS_PLATFORM == "win32":
+        startupinfo = subprocess.STARTUPINFO()
+        try:
+            startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
+        except AttributeError:
+            startupinfo.dwFlags |= subprocess._subprocess.STARTF_USESHOWWINDOW
+
+    env = os.environ.copy()
+    if 'xld' in split_cmd:
+        env['PATH'] += os.pathsep + '/Applications'
+    elif headphones.CONFIG.CUE_SPLIT_FLAC_PATH:
+        env['PATH'] += os.pathsep + headphones.CONFIG.CUE_SPLIT_FLAC_PATH
+
+    process = subprocess.Popen(split_cmd, startupinfo=startupinfo,
+                               stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE, env=env)
+    stdout, stderr = process.communicate()
+
+    if process.returncode:
+        logger.error('Split failed for %s', split_file.decode(headphones.SYS_ENCODING, 'replace'))
+        out = stdout if stdout else stderr
+        logger.error('Error details: %s', out.decode(headphones.SYS_ENCODING, 'replace'))
+        return False
+    else:
+        logger.info('Split success %s', split_file.decode(headphones.SYS_ENCODING, 'replace'))
+        return True
+
+
+def check_list(list, ignore=0):
+    '''Checks a list for None elements. If list have None (after ignore index) then it should pass only if all elements
+    are None threreafter. Returns a tuple without the None entries.'''
+
+    if ignore:
+        try:
+            list[int(ignore)]
+        except:
+            raise ValueError('non-integer ignore index or ignore index not in list')
+
+    list1 = list[:ignore]
+    list2 = list[ignore:]
+
+    try:
+        first_none = list2.index(None)
+    except:
+        return tuple(list1 + list2)
+
+    for i in range(first_none, len(list2)):
+        if list2[i]:
+            raise ValueError('non-None entry after None entry in list at index {0}'.format(i))
+
+    while True:
+        list2.remove(None)
+        try:
+            list2.index(None)
+        except:
+            break
+
+    return tuple(list1 + list2)
+
+
+def trim_cue_entry(string):
+    '''Removes leading and trailing "s.'''
+    if string[0] == '"' and string[-1] == '"':
+        string = string[1:-1]
+    return string
+
+
+def int_to_str(value, length=2):
+    '''Converts integer to string eg 3 to "03"'''
+    try:
+        int(value)
+    except:
+        raise ValueError('expected an integer value')
+
+    content = str(value)
+    while len(content) < length:
+        content = '0' + content
+    return content
+
+
+class Directory:
+    def __init__(self, path):
+        self.path = path
+        self.name = os.path.split(self.path)[-1]
+        self.content = []
+        self.update()
+
+    def filter(self, classname):
+        content = []
+        for c in self.content:
+            if c.__class__.__name__ == classname:
+                content.append(c)
+        return content
+
+    def tracks(self, ext=None, split=False):
+        content = []
+        for c in self.content:
+            ext_match = False
+            if c.__class__.__name__ == 'WaveFile':
+                if not ext or (ext and ext == c.name_ext):
+                    ext_match = True
+                if ext_match and c.track_nr:
+                    if not split or (split and c.split_file):
+                        content.append(c)
+        return content
+
+    def update(self):
+        def check_match(filename):
+            for i in self.content:
+                if i.name == filename:
+                    return True
+            return False
+
+        def identify_track_number(filename):
+            if 'split-track' in filename:
+                search = re.search('split-track(\d\d)', filename)
+                if search:
+                    n = int(search.group(1))
+                    if n:
+                        return n
+            for n in range(0, 100):
+                search = re.search(int_to_str(n), filename)
+                if search:
+                    # TODO: not part of other value such as year
+                    return n
+
+        list_dir = glob.glob1(self.path, '*')
+
+        # TODO: for some reason removes only one file
+        rem_list = []
+        for i in self.content:
+            if i.name not in list_dir:
+                rem_list.append(i)
+        for i in rem_list:
+            self.content.remove(i)
+
+        for i in list_dir:
+            if not check_match(i):
+                # music file
+                if os.path.splitext(i)[-1] in WAVE_FILE_TYPE_BY_EXTENSION.keys():
+                    track_nr = identify_track_number(i)
+                    if track_nr:
+                        self.content.append(WaveFile(self.path + os.sep + i, track_nr=track_nr))
+                    else:
+                        self.content.append(WaveFile(self.path + os.sep + i))
+
+                # cue file
+                elif os.path.splitext(i)[-1] == '.cue':
+                    self.content.append(CueFile(self.path + os.sep + i))
+
+                # meta file
+                elif i == ALBUM_META_FILE_NAME:
+                    self.content.append(MetaFile(self.path + os.sep + i))
+
+                # directory
+                elif os.path.isdir(i):
+                    self.content.append(Directory(self.path + os.sep + i))
+
+                else:
+                    self.content.append(File(self.path + os.sep + i))
+
+
+class File(object):
+    def __init__(self, path):
+        self.path = path
+        self.name = os.path.split(self.path)[-1]
+
+        self.name_name = ''.join(os.path.splitext(self.name)[:-1])
+        self.name_ext = os.path.splitext(self.name)[-1]
+        self.split_file = True if self.name_name[:11] == 'split-track' else False
+
+    def get_name(self, ext=True, cmd=False):
+
+        if ext is True:
+            content = self.name
+        elif ext is False:
+            content = self.name_name
+        elif ext[0] == '.':
+            content = self.name_name + ext
+        else:
+            raise ValueError('ext parameter error')
+
+        if cmd:
+            content = content.replace(' ', '\ ')
+
+        return content
+
+
+class CueFile(File):
+    def __init__(self, path):
+
+        def header_parser():
+            global line_content
+            c = self.content.splitlines()
+            header_dict = {}
+            # remaining_headers = CUE_HEADER
+            remaining_headers = copy.copy(CUE_HEADER)
+            line_index = 0
+            match = True
+            while match:
+                match = False
+                saved_match = None
+                line_content = c[line_index]
+                for e in remaining_headers:
+                    search_result = re.search(remaining_headers[e], line_content, re.I)
+                    if search_result:
+                        search_content = trim_cue_entry(search_result.group(1))
+                        header_dict[e] = search_content
+                        saved_match = e
+                        match = True
+                        line_index += 1
+                if saved_match:
+                    del remaining_headers[saved_match]
+            return header_dict, line_index
+
+        def track_parser(start_line):
+            c = self.content.splitlines()
+            line_index = start_line
+            line_content = c[line_index]
+            search_result = re.search(CUE_TRACK, line_content, re.I)
+            if not search_result:
+                raise ValueError(
+                    'inconsistent CUE sheet, TRACK expected at line {0}'.format(line_index + 1))
+            track_nr = int(search_result.group(1))
+            line_index += 1
+            next_track = False
+            track_meta = {}
+            # we make room for future indexes
+            track_meta['index'] = [None for m in range(100)]
+
+            while not next_track:
+                if line_index < len(c):
+                    line_content = c[line_index]
+
+                    artist_search = re.search(CUE_TRACK_INFO['artist'], line_content, re.I)
+                    title_search = re.search(CUE_TRACK_INFO['title'], line_content, re.I)
+                    isrc_search = re.search(CUE_TRACK_INFO['isrc'], line_content, re.I)
+                    index_search = re.search(CUE_TRACK_INFO['index'], line_content, re.I)
+
+                    if artist_search:
+                        if trim_cue_entry(artist_search.group(1)) != self.header['artist']:
+                            track_meta['artist'] = trim_cue_entry(artist_search.group(1))
+                        line_index += 1
+                    elif title_search:
+                        track_meta['title'] = trim_cue_entry(title_search.group(1))
+                        line_index += 1
+                    elif isrc_search:
+                        track_meta['isrc'] = trim_cue_entry(isrc_search.group(1))
+                        line_index += 1
+                    elif index_search:
+                        track_meta['index'][int(index_search.group(1))] = index_search.group(2)
+                        line_index += 1
+                    elif re.search(CUE_TRACK, line_content, re.I):
+                        next_track = True
+                    elif line_index == len(c) - 1 and not line_content:
+                        # last line is empty
+                        line_index += 1
+                    elif re.search('FLAGS DCP$', line_content, re.I):
+                        track_meta['dcpflag'] = True
+                        line_index += 1
+                    else:
+                        raise ValueError(
+                            'unknown entry in track error, line {0}'.format(line_index + 1))
+                else:
+                    next_track = True
+
+            track_meta['index'] = check_list(track_meta['index'], ignore=1)
+
+            return track_nr, track_meta, line_index
+
+        super(CueFile, self).__init__(path)
+
+        try:
+            with open(self.name) as cue_file:
+                self.content = cue_file.read()
+        except:
+            self.content = None
+
+        if not self.content:
+            try:
+                with open(self.name, encoding="cp1252") as cue_file:
+                    self.content = cue_file.read()
+            except:
+                raise ValueError('Cant encode CUE Sheet.')
+
+        if self.content[0] == u'\ufeff':
+            self.content = self.content[1:]
+
+        header = header_parser()
+
+        self.header = header[0]
+
+        line_index = header[1]
+
+        # we make room for tracks
+        tracks = [None for m in range(100)]
+
+        while line_index < len(self.content.splitlines()):
+            parsed_track = track_parser(line_index)
+            line_index = parsed_track[2]
+            tracks[parsed_track[0]] = parsed_track[1]
+
+        self.tracks = check_list(tracks, ignore=1)
+
+    def get_meta(self):
+        content = ''
+        for i in ALBUM_META_ALBUM_BY_CUE:
+            if self.header.get(i):
+                content += i + '\t' + self.header[i] + '\n'
+            else:
+                content += i + '\t' + '\n'
+
+        for i in range(len(self.tracks)):
+            if self.tracks[i]:
+                if self.tracks[i].get('artist'):
+                    content += 'track' + int_to_str(i) + 'artist' + '\t' + self.tracks[i].get(
+                        'artist') + '\n'
+                if self.tracks[i].get('title'):
+                    content += 'track' + int_to_str(i) + 'title' + '\t' + self.tracks[i].get(
+                        'title') + '\n'
+        return content
+
+    def htoa(self):
+        '''Returns true if Hidden Track exists.'''
+        if int(self.tracks[1]['index'][1][-5:-3]) >= HTOA_LENGTH_TRIGGER:
+            return True
+        return False
+
+    def breakpoints(self):
+        '''Returns track break points. Identical as CUETools' cuebreakpoints, with the exception of my standards for HTOA.'''
+        content = ''
+        for t in range(len(self.tracks)):
+            if t == 1 and not self.htoa():
+                content += ''
+            elif t >= 1:
+                t_index = self.tracks[t]['index']
+                content += t_index[1]
+                if t < (len(self.tracks) - 1):
+                    content += '\n'
+        return content
+
+
+class MetaFile(File):
+    def __init__(self, path):
+        super(MetaFile, self).__init__(path)
+        with open(self.path) as meta_file:
+            self.rawcontent = meta_file.read()
+
+        content = {}
+        content['tracks'] = [None for m in range(100)]
+
+        for l in self.rawcontent.splitlines():
+            parsed_line = re.search('^(.+?)\t(.+?)$', l)
+            if parsed_line:
+                if parsed_line.group(1)[:5] == 'track':
+                    parsed_track = re.search('^track(\d\d)(.+?)$', parsed_line.group(1))
+                    if not parsed_track:
+                        raise ValueError('Syntax error in album meta file')
+                    if not content['tracks'][int(parsed_track.group(1))]:
+                        content['tracks'][int(parsed_track.group(1))] = dict()
+                    content['tracks'][int(parsed_track.group(1))][
+                        parsed_track.group(2)] = parsed_line.group(2)
+                else:
+                    content[parsed_line.group(1)] = parsed_line.group(2)
+
+        content['tracks'] = check_list(content['tracks'], ignore=1)
+
+        self.content = content
+
+    def flac_tags(self, track_nr):
+        common_tags = dict()
+        freeform_tags = dict()
+
+        # common flac tags
+        common_tags['artist'] = self.content['artist']
+        common_tags['album'] = self.content['title']
+        common_tags['title'] = self.content['tracks'][track_nr]['title']
+        common_tags['tracknumber'] = str(track_nr)
+        common_tags['tracktotal'] = str(len(self.content['tracks']) - 1)
+        if 'date' in self.content:
+            common_tags['date'] = self.content['date']
+        if 'genre' in CUE_META.content:
+            common_tags['genre'] = CUE_META.content['genre']
+
+        # freeform tags
+        # freeform_tags['country'] = self.content['country']
+        # freeform_tags['releasedate'] = self.content['releasedate']
+
+        return common_tags, freeform_tags
+
+    def folders(self):
+        artist = self.content['artist']
+        album = self.content['date'] + ' - ' + self.content['title'] + ' (' + self.content[
+            'label'] + ' - ' + self.content['catalog'] + ')'
+        return artist, album
+
+    def complete(self):
+        '''Check MetaFile for containing all data'''
+        self.__init__(self.path)
+        for l in self.rawcontent.splitlines():
+            if re.search('^[0-9A-Za-z]+?\t$', l):
+                return False
+        return True
+
+    def count_tracks(self):
+        '''Returns tracks count'''
+        return len(self.content['tracks']) - self.content['tracks'].count(None)
+
+
+class WaveFile(File):
+    def __init__(self, path, track_nr=None):
+        super(WaveFile, self).__init__(path)
+
+        self.track_nr = track_nr
+        self.type = WAVE_FILE_TYPE_BY_EXTENSION[self.name_ext]
+
+    def filename(self, ext=None, cmd=False):
+        title = CUE_META.content['tracks'][self.track_nr]['title']
+
+        if ext:
+            if ext[0] != '.':
+                ext = '.' + ext
+        else:
+            ext = self.name_ext
+
+        f_name = int_to_str(self.track_nr) + ' - ' + title + ext
+
+        if cmd:
+            f_name = f_name.replace(' ', '\ ')
+
+        f_name = f_name.replace('!', '')
+        f_name = f_name.replace('?', '')
+        f_name = f_name.replace('/', ';')
+
+        return f_name
+
+    def tag(self):
+        if self.type == 'Free Lossless Audio Codec':
+            f = FLAC(self.name)
+            tags = CUE_META.flac_tags(self.track_nr)
+            for t in tags[0]:
+                f[t] = tags[0][t]
+            f.save()
+
+    def mutagen(self):
+        if self.type == 'Free Lossless Audio Codec':
+            return FLAC(self.name)
+
+
+def split(albumpath):
+    global CUE_META
+    os.chdir(albumpath)
+    base_dir = Directory(os.getcwd())
+
+    cue = None
+    wave = None
+
+    # determining correct cue file
+    # if perfect match found
+    for _cue in base_dir.filter('CueFile'):
+        for _wave in base_dir.filter('WaveFile'):
+            if _cue.header['file'] == _wave.name:
+                logger.info('CUE Sheet found: %s', _cue.name)
+                logger.info('Music file found: %s', _wave.name)
+                cue = _cue
+                wave = _wave
+    # if no perfect match found then try without extensions
+    if not cue and not wave:
+        logger.info('No match for music files, trying to match without extensions...')
+        for _cue in base_dir.filter('CueFile'):
+            for _wave in base_dir.filter('WaveFile'):
+                if ''.join(os.path.splitext(_cue.header['file'])[:-1]) == _wave.name_name:
+                    logger.info('Possible CUE Sheet found: %s', _cue.name)
+                    logger.info('CUE Sheet refers music file: %s', _cue.header['file'])
+                    logger.info('Possible Music file found: %s', _wave.name)
+                    cue = _cue
+                    wave = _wave
+                    cue.header['file'] = wave.name
+    # if still no match then raise an exception
+    if not cue and not wave:
+        raise ValueError('No music file match found!')
+
+    # Split with xld or shntool
+    splitter = 'shntool'
+    xldprofile = None
+
+    # use xld profile to split cue
+    if headphones.CONFIG.ENCODER == 'xld' and headphones.CONFIG.MUSIC_ENCODER and headphones.CONFIG.XLDPROFILE:
+        import getXldProfile
+        xldprofile, xldformat, _ = getXldProfile.getXldProfile(headphones.CONFIG.XLDPROFILE)
+        if not xldformat:
+            raise ValueError(
+                'Details for xld profile "%s" not found, cannot split cue' % (xldprofile))
+        else:
+            if headphones.CONFIG.ENCODERFOLDER:
+                splitter = os.path.join(headphones.CONFIG.ENCODERFOLDER, 'xld')
+            else:
+                splitter = 'xld'
+    # use standard xld command to split cue
+    elif sys.platform == 'darwin':
+        splitter = 'xld'
+        if not check_splitter(splitter):
+            splitter = 'shntool'
+
+    if splitter == 'shntool' and not check_splitter(splitter):
+        raise ValueError('Command not found, ensure shntool or xld installed')
+
+    # Determine if file can be split
+    if wave.name_ext not in WAVE_FILE_TYPE_BY_EXTENSION.keys():
+        raise ValueError('Cannot split, audio file has unsupported extension')
+
+    # Split with xld
+    if 'xld' in splitter:
+        cmd = [splitter]
+        cmd.extend([wave.name])
+        cmd.extend(['-c'])
+        cmd.extend([cue.name])
+        if xldprofile:
+            cmd.extend(['--profile'])
+            cmd.extend([xldprofile])
+        else:
+            cmd.extend(['-f'])
+            cmd.extend(['flac'])
+        cmd.extend(['-o'])
+        cmd.extend([base_dir.path])
+        split = split_baby(wave.name, cmd)
+    else:
+
+        # Split with shntool
+
+        # generate temporary metafile describing the cue
+        with open(ALBUM_META_FILE_NAME, mode='w') as meta_file:
+            meta_file.write(cue.get_meta())
+        base_dir.content.append(MetaFile(os.path.abspath(ALBUM_META_FILE_NAME)))
+        # check metafile for completeness
+        if not base_dir.filter('MetaFile'):
+            raise ValueError('Cue Meta file {0} missing!'.format(ALBUM_META_FILE_NAME))
+        else:
+            CUE_META = base_dir.filter('MetaFile')[0]
+
+        with open(SPLIT_FILE_NAME, mode='w') as split_file:
+            split_file.write(cue.breakpoints())
+
+        if headphones.CONFIG.CUE_SPLIT_SHNTOOL_PATH:
+            cmd = [os.path.join(headphones.CONFIG.CUE_SPLIT_SHNTOOL_PATH, 'shntool')]
+        else:
+            cmd = ['shntool']
+
+        cmd.extend(['split'])
+        cmd.extend(['-f'])
+        cmd.extend([SPLIT_FILE_NAME])
+        cmd.extend(['-o'])
+        cmd.extend([wave.name_ext.lstrip('.')])
+        cmd.extend([wave.name])
+        split = split_baby(wave.name, cmd)
+        os.remove(SPLIT_FILE_NAME)
+        base_dir.update()
+
+        # tag FLAC files
+        if split and CUE_META.count_tracks() == len(base_dir.tracks(ext='.flac', split=True)):
+            for t in base_dir.tracks(ext='.flac', split=True):
+                logger.info('Tagging %s...', t.name)
+                t.tag()
+
+        # rename files
+        if split and CUE_META.count_tracks() == len(base_dir.tracks(ext=wave.name_ext, split=True)):
+            for t in base_dir.tracks(ext=wave.name_ext, split=True):
+                if t.name != t.filename():
+                    logger.info('Renaming %s to %s...', t.name, t.filename())
+                    os.rename(t.name, t.filename())
+
+        os.remove(ALBUM_META_FILE_NAME)
+
+    if not split:
+        raise ValueError('Failed to split, check logs')
+    else:
+        # Rename original file
+        os.rename(wave.name, wave.name + '.original')
+        return True

headphones/db.py

@@ -13,60 +13,58 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-#####################################
-## Stolen from Sick-Beard's db.py  ##
-#####################################
+###################################
+# Stolen from Sick-Beard's db.py  #
+###################################
 
 from __future__ import with_statement
 
-import os
 import sqlite3
-import threading
-import time
 
+import os
 import headphones
-
 from headphones import logger
 
-def dbFilename(filename="headphones.db"):
 
+def dbFilename(filename="headphones.db"):
     return os.path.join(headphones.DATA_DIR, filename)
 
+
 def getCacheSize():
-    #this will protect against typecasting problems produced by empty string and None settings
-    if not headphones.CACHE_SIZEMB:
-        #sqlite will work with this (very slowly)
+    # this will protect against typecasting problems produced by empty string and None settings
+    if not headphones.CONFIG.CACHE_SIZEMB:
+        # sqlite will work with this (very slowly)
         return 0
-    return int(headphones.CACHE_SIZEMB)
+    return int(headphones.CONFIG.CACHE_SIZEMB)
+
 
 class DBConnection:
-
     def __init__(self, filename="headphones.db"):
 
         self.filename = filename
         self.connection = sqlite3.connect(dbFilename(filename), timeout=20)
-        #don't wait for the disk to finish writing
+        # don't wait for the disk to finish writing
         self.connection.execute("PRAGMA synchronous = OFF")
-        #journal disabled since we never do rollbacks
-        self.connection.execute("PRAGMA journal_mode = %s" % headphones.JOURNAL_MODE)
-        #64mb of cache memory,probably need to make it user configurable
-        self.connection.execute("PRAGMA cache_size=-%s" % (getCacheSize()*1024))
+        # journal disabled since we never do rollbacks
+        self.connection.execute("PRAGMA journal_mode = %s" % headphones.CONFIG.JOURNAL_MODE)
+        # 64mb of cache memory,probably need to make it user configurable
+        self.connection.execute("PRAGMA cache_size=-%s" % (getCacheSize() * 1024))
         self.connection.row_factory = sqlite3.Row
 
     def action(self, query, args=None):
 
-        if query == None:
+        if query is None:
             return
 
         sqlResult = None
-        
+
         try:
             with self.connection as c:
-                if args == None:
+                if args is None:
                     sqlResult = c.execute(query)
                 else:
                     sqlResult = c.execute(query, args)
-                    
+
         except sqlite3.OperationalError, e:
             if "unable to open database file" in e.message or "database is locked" in e.message:
                 logger.warn('Database Error: %s', e)
@@ -77,29 +75,37 @@ class DBConnection:
         except sqlite3.DatabaseError, e:
             logger.error('Fatal Error executing %s :: %s', query, e)
             raise
-        
+
         return sqlResult
 
     def select(self, query, args=None):
 
         sqlResults = self.action(query, args).fetchall()
-        
-        if sqlResults == None or sqlResults == [None]:
+
+        if sqlResults is None or sqlResults == [None]:
             return []
 
         return sqlResults
 
     def upsert(self, tableName, valueDict, keyDict):
 
+        def genParams(myDict):
+            return [x + " = ?" for x in myDict.keys()]
+
         changesBefore = self.connection.total_changes
 
-        genParams = lambda myDict : [x + " = ?" for x in myDict.keys()]
+        update_query = "UPDATE " + tableName + " SET " + ", ".join(
+            genParams(valueDict)) + " WHERE " + " AND ".join(genParams(keyDict))
 
-        query = "UPDATE "+tableName+" SET " + ", ".join(genParams(valueDict)) + " WHERE " + " AND ".join(genParams(keyDict))
-
-        self.action(query, valueDict.values() + keyDict.values())
+        self.action(update_query, valueDict.values() + keyDict.values())
 
         if self.connection.total_changes == changesBefore:
-            query = "INSERT INTO "+tableName+" (" + ", ".join(valueDict.keys() + keyDict.keys()) + ")" + \
-                        " VALUES (" + ", ".join(["?"] * len(valueDict.keys() + keyDict.keys())) + ")"
-            self.action(query, valueDict.values() + keyDict.values())
+            insert_query = (
+                "INSERT INTO " + tableName + " (" + ", ".join(
+                    valueDict.keys() + keyDict.keys()) + ")" +
+                " VALUES (" + ", ".join(["?"] * len(valueDict.keys() + keyDict.keys())) + ")"
+            )
+            try:
+                self.action(insert_query, valueDict.values() + keyDict.values())
+            except sqlite3.IntegrityError:
+                logger.info('Queries failed: %s and %s', update_query, insert_query)

headphones/exceptions.py

@@ -13,11 +13,13 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
+
 class HeadphonesException(Exception):
     """
     Generic Headphones Exception - should never be thrown, only subclassed
     """
 
+
 class NewzbinAPIThrottled(HeadphonesException):
     """
     Newzbin has throttled us, deal with it

headphones/getXldProfile.py

@@ -1,32 +1,31 @@
 import os.path
-import plistlib
-import sys
-import xml.parsers.expat as expat
-import commands
+
+import biplist
 from headphones import logger
 
+
 def getXldProfile(xldProfile):
     xldProfileNotFound = xldProfile
-    expandedPath = os.path.expanduser('~/Library/Preferences/jp.tmkk.XLD.plist')
+
+    expanded = os.path.expanduser('~/Library/Preferences/jp.tmkk.XLD.plist')
+    if not os.path.isfile(expanded):
+        logger.warn("Could not find xld preferences at: %s", expanded)
+        return (xldProfileNotFound, None, None)
+
+    # Get xld preferences plist
     try:
-        preferences = plistlib.Plist.fromFile(expandedPath)
-    except (expat.ExpatError):
-        os.system("/usr/bin/plutil -convert xml1 %s" % expandedPath )
-        try:
-            preferences = plistlib.Plist.fromFile(expandedPath)
-        except (ImportError):
-            os.system("/usr/bin/plutil -convert binary1 %s" % expandedPath )
-            logger.info('The plist at "%s" has a date in it, and therefore is not useable.' % expandedPath)
-            return(xldProfileNotFound, None, None)
-    except (ImportError):
-        logger.info('The plist at "%s" has a date in it, and therefore is not useable.' % expandedPath)
-    except:
-        logger.info('Unexpected error:', sys.exc_info()[0])
-        return(xldProfileNotFound, None, None)
+        preferences = biplist.readPlist(expanded)
+    except (biplist.InvalidPlistException, biplist.NotBinaryPlistException), e:
+        logger.error("Error reading xld preferences plist: %s", e)
+        return (xldProfileNotFound, None, None)
+
+    if not isinstance(preferences, dict):
+        logger.error("Error reading xld preferences plist, not a dict: %r", preferences)
+        return (xldProfileNotFound, None, None)
+
+    profiles = preferences.get('Profiles', [])  # pylint:disable=E1103
 
     xldProfile = xldProfile.lower()
-    profiles = preferences.get('Profiles')
-
     for profile in profiles:
 
         profilename = profile.get('XLDProfileManager_ProfileName')
@@ -176,6 +175,6 @@ def getXldProfile(xldProfile):
             if xldFormat and not xldBitrate:
                 xldBitrate = 400
 
-            return(xldProfileForCmd, xldFormat, xldBitrate)
+            return (xldProfileForCmd, xldFormat, xldBitrate)
 
-    return(xldProfileNotFound, None, None)
+    return (xldProfileNotFound, None, None)

headphones/helpers.py

@@ -13,17 +13,19 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
-import re
-import time
-import shutil
-import datetime
-import headphones
+from operator import itemgetter
 import unicodedata
+import datetime
+import shutil
+import time
 import sys
 
-from operator import itemgetter
+import fnmatch
+import re
+import os
 from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
+import headphones
+
 
 # Modified from https://github.com/Verrus/beets-plugin-featInTitle
 RE_FEATURING = re.compile(r"[fF]t\.|[fF]eaturing|[fF]eat\.|\b[wW]ith\b|&|vs\.")
@@ -31,8 +33,11 @@ RE_FEATURING = re.compile(r"[fF]t\.|[fF]eaturing|[fF]eat\.|\b[wW]ith\b|&|vs\.")
 RE_CD_ALBUM = re.compile(r"\(?((CD|disc)\s*[0-9]+)\)?", re.I)
 RE_CD = re.compile(r"^(CD|dics)\s*[0-9]+$", re.I)
 
+
 def multikeysort(items, columns):
-    comparers = [ ((itemgetter(col[1:].strip()), -1) if col.startswith('-') else (itemgetter(col.strip()), 1)) for col in columns]
+    comparers = [
+        ((itemgetter(col[1:].strip()), -1) if col.startswith('-') else (itemgetter(col.strip()), 1))
+        for col in columns]
 
     def comparer(left, right):
         for fn, mult in comparers:
@@ -44,53 +49,56 @@ def multikeysort(items, columns):
 
     return sorted(items, cmp=comparer)
 
+
 def checked(variable):
     if variable:
         return 'Checked'
     else:
         return ''
 
-def radio(variable, pos):
 
+def radio(variable, pos):
     if variable == pos:
         return 'Checked'
     else:
         return ''
 
+
 def latinToAscii(unicrap):
     """
     From couch potato
     """
-    xlate = {0xc0:'A', 0xc1:'A', 0xc2:'A', 0xc3:'A', 0xc4:'A', 0xc5:'A',
-        0xc6:'Ae', 0xc7:'C',
-        0xc8:'E', 0xc9:'E', 0xca:'E', 0xcb:'E', 0x86:'e',
-        0xcc:'I', 0xcd:'I', 0xce:'I', 0xcf:'I',
-        0xd0:'Th', 0xd1:'N',
-        0xd2:'O', 0xd3:'O', 0xd4:'O', 0xd5:'O', 0xd6:'O', 0xd8:'O',
-        0xd9:'U', 0xda:'U', 0xdb:'U', 0xdc:'U',
-        0xdd:'Y', 0xde:'th', 0xdf:'ss',
-        0xe0:'a', 0xe1:'a', 0xe2:'a', 0xe3:'a', 0xe4:'a', 0xe5:'a',
-        0xe6:'ae', 0xe7:'c',
-        0xe8:'e', 0xe9:'e', 0xea:'e', 0xeb:'e', 0x0259:'e',
-        0xec:'i', 0xed:'i', 0xee:'i', 0xef:'i',
-        0xf0:'th', 0xf1:'n',
-        0xf2:'o', 0xf3:'o', 0xf4:'o', 0xf5:'o', 0xf6:'o', 0xf8:'o',
-        0xf9:'u', 0xfa:'u', 0xfb:'u', 0xfc:'u',
-        0xfd:'y', 0xfe:'th', 0xff:'y',
-        0xa1:'!', 0xa2:'{cent}', 0xa3:'{pound}', 0xa4:'{currency}',
-        0xa5:'{yen}', 0xa6:'|', 0xa7:'{section}', 0xa8:'{umlaut}',
-        0xa9:'{C}', 0xaa:'{^a}', 0xab:'<<', 0xac:'{not}',
-        0xad:'-', 0xae:'{R}', 0xaf:'_', 0xb0:'{degrees}',
-        0xb1:'{+/-}', 0xb2:'{^2}', 0xb3:'{^3}', 0xb4:"'",
-        0xb5:'{micro}', 0xb6:'{paragraph}', 0xb7:'*', 0xb8:'{cedilla}',
-        0xb9:'{^1}', 0xba:'{^o}', 0xbb:'>>',
-        0xbc:'{1/4}', 0xbd:'{1/2}', 0xbe:'{3/4}', 0xbf:'?',
-        0xd7:'*', 0xf7:'/'
-        }
+    xlate = {
+        0xc0: 'A', 0xc1: 'A', 0xc2: 'A', 0xc3: 'A', 0xc4: 'A', 0xc5: 'A',
+        0xc6: 'Ae', 0xc7: 'C',
+        0xc8: 'E', 0xc9: 'E', 0xca: 'E', 0xcb: 'E', 0x86: 'e', 0x39e: 'E',
+        0xcc: 'I', 0xcd: 'I', 0xce: 'I', 0xcf: 'I',
+        0xd0: 'Th', 0xd1: 'N',
+        0xd2: 'O', 0xd3: 'O', 0xd4: 'O', 0xd5: 'O', 0xd6: 'O', 0xd8: 'O',
+        0xd9: 'U', 0xda: 'U', 0xdb: 'U', 0xdc: 'U',
+        0xdd: 'Y', 0xde: 'th', 0xdf: 'ss',
+        0xe0: 'a', 0xe1: 'a', 0xe2: 'a', 0xe3: 'a', 0xe4: 'a', 0xe5: 'a',
+        0xe6: 'ae', 0xe7: 'c',
+        0xe8: 'e', 0xe9: 'e', 0xea: 'e', 0xeb: 'e', 0x0259: 'e',
+        0xec: 'i', 0xed: 'i', 0xee: 'i', 0xef: 'i',
+        0xf0: 'th', 0xf1: 'n',
+        0xf2: 'o', 0xf3: 'o', 0xf4: 'o', 0xf5: 'o', 0xf6: 'o', 0xf8: 'o',
+        0xf9: 'u', 0xfa: 'u', 0xfb: 'u', 0xfc: 'u',
+        0xfd: 'y', 0xfe: 'th', 0xff: 'y',
+        0xa1: '!', 0xa2: '{cent}', 0xa3: '{pound}', 0xa4: '{currency}',
+        0xa5: '{yen}', 0xa6: '|', 0xa7: '{section}', 0xa8: '{umlaut}',
+        0xa9: '{C}', 0xaa: '{^a}', 0xab: '<<', 0xac: '{not}',
+        0xad: '-', 0xae: '{R}', 0xaf: '_', 0xb0: '{degrees}',
+        0xb1: '{+/-}', 0xb2: '{^2}', 0xb3: '{^3}', 0xb4: "'",
+        0xb5: '{micro}', 0xb6: '{paragraph}', 0xb7: '*', 0xb8: '{cedilla}',
+        0xb9: '{^1}', 0xba: '{^o}', 0xbb: '>>',
+        0xbc: '{1/4}', 0xbd: '{1/2}', 0xbe: '{3/4}', 0xbf: '?',
+        0xd7: '*', 0xf7: '/'
+    }
 
     r = ''
     for i in unicrap:
-        if xlate.has_key(ord(i)):
+        if ord(i) in xlate:
             r += xlate[ord(i)]
         elif ord(i) >= 0x80:
             pass
@@ -98,9 +106,9 @@ def latinToAscii(unicrap):
             r += str(i)
     return r
 
-def convert_milliseconds(ms):
 
-    seconds = ms/1000
+def convert_milliseconds(ms):
+    seconds = ms / 1000
     gmtime = time.gmtime(seconds)
     if seconds > 3600:
         minutes = time.strftime("%H:%M:%S", gmtime)
@@ -109,8 +117,8 @@ def convert_milliseconds(ms):
 
     return minutes
 
-def convert_seconds(s):
 
+def convert_seconds(s):
     gmtime = time.gmtime(s)
     if s > 3600:
         minutes = time.strftime("%H:%M:%S", gmtime)
@@ -119,50 +127,61 @@ def convert_seconds(s):
 
     return minutes
 
+
 def today():
     today = datetime.date.today()
     yyyymmdd = datetime.date.isoformat(today)
     return yyyymmdd
 
+
 def now():
     now = datetime.datetime.now()
     return now.strftime("%Y-%m-%d %H:%M:%S")
 
-def get_age(date):
 
+def get_age(date):
     try:
         split_date = date.split('-')
     except:
         return False
 
     try:
-        days_old = int(split_date[0])*365 + int(split_date[1])*30 + int(split_date[2])
-    except IndexError:
+        days_old = int(split_date[0]) * 365 + int(split_date[1]) * 30 + int(split_date[2])
+    except (IndexError, ValueError):
         days_old = False
 
     return days_old
 
-def bytes_to_mb(bytes):
 
-    mb = int(bytes)/1048576
+def bytes_to_mb(bytes):
+    mb = int(bytes) / 1048576
     size = '%.1f MB' % mb
     return size
 
+
 def mb_to_bytes(mb_str):
     result = re.search('^(\d+(?:\.\d+)?)\s?(?:mb)?', mb_str, flags=re.I)
     if result:
-        return int(float(result.group(1))*1048576)
+        return int(float(result.group(1)) * 1048576)
+
 
 def piratesize(size):
     split = size.split(" ")
     factor = float(split[0])
-    unit = split[1]
-    if unit == 'MiB':
+    unit = split[1].upper()
+
+    if unit == 'MIB':
         size = factor * 1048576
-    elif unit == 'GiB':
+    elif unit == 'MB':
+        size = factor * 1000000
+    elif unit == 'GIB':
         size = factor * 1073741824
-    elif unit == 'KiB':
+    elif unit == 'GB':
+        size = factor * 1000000000
+    elif unit == 'KIB':
         size = factor * 1024
+    elif unit == 'KB':
+        size = factor * 1000
     elif unit == "B":
         size = factor
     else:
@@ -170,20 +189,24 @@ def piratesize(size):
 
     return size
 
-def replace_all(text, dic, normalize=False):
 
+def replace_all(text, dic, normalize=False):
     if not text:
         return ''
 
     for i, j in dic.iteritems():
         if normalize:
-            if sys.platform == 'darwin':
-                j = unicodedata.normalize('NFD', j)
-            else:
-                j = unicodedata.normalize('NFC', j)
+            try:
+                if sys.platform == 'darwin':
+                    j = unicodedata.normalize('NFD', j)
+                else:
+                    j = unicodedata.normalize('NFC', j)
+            except TypeError:
+                j = unicodedata.normalize('NFC', j.decode(headphones.SYS_ENCODING, 'replace'))
         text = text.replace(i, j)
     return text
 
+
 def replace_illegal_chars(string, type="file"):
     if type == "file":
         string = re.sub('[\?"*:|<>/]', '_', string)
@@ -192,15 +215,16 @@ def replace_illegal_chars(string, type="file"):
 
     return string
 
-def cleanName(string):
 
+def cleanName(string):
     pass1 = latinToAscii(string).lower()
-    out_string = re.sub('[\.\-\/\!\@\#\$\%\^\&\*\(\)\+\-\"\'\,\;\:\[\]\{\}\<\>\=\_]', '', pass1).encode('utf-8')
+    out_string = re.sub('[\.\-\/\!\@\#\$\%\^\&\*\(\)\+\-\"\'\,\;\:\[\]\{\}\<\>\=\_]', '',
+                        pass1).encode('utf-8')
 
     return out_string
 
-def cleanTitle(title):
 
+def cleanTitle(title):
     title = re.sub('[\.\-\/\_]', ' ', title).lower()
 
     # Strip out extra whitespace
@@ -210,6 +234,7 @@ def cleanTitle(title):
 
     return title
 
+
 def split_path(f):
     """
     Split a path into components, starting with the drive letter (if any). Given
@@ -219,7 +244,7 @@ def split_path(f):
     components = []
     drive, path = os.path.splitdrive(f)
 
-    # Stip the folder from the path, iterate until nothing is left
+    # Strip the folder from the path, iterate until nothing is left
     while True:
         path, folder = os.path.split(path)
 
@@ -241,6 +266,7 @@ def split_path(f):
     # Done
     return components
 
+
 def expand_subfolders(f):
     """
     Try to expand a given folder and search for subfolders containing media
@@ -269,7 +295,7 @@ def expand_subfolders(f):
         return
 
     # Split into path components
-    media_folders = [ split_path(media_folder) for media_folder in media_folders ]
+    media_folders = [split_path(media_folder) for media_folder in media_folders]
 
     # Correct folder endings such as CD1 etc.
     for index, media_folder in enumerate(media_folders):
@@ -277,25 +303,31 @@ def expand_subfolders(f):
             media_folders[index] = media_folders[index][:-1]
 
     # Verify the result by computing path depth relative to root.
-    path_depths = [ len(media_folder) for media_folder in media_folders ]
+    path_depths = [len(media_folder) for media_folder in media_folders]
     difference = max(path_depths) - min(path_depths)
 
     if difference > 0:
-        logger.info("Found %d media folders, but depth difference between lowest and deepest media folder is %d (expected zero). If this is a discography or a collection of albums, make sure albums are per folder.", len(media_folders), difference)
+        logger.info(
+            "Found %d media folders, but depth difference between lowest and deepest media folder is %d (expected zero). If this is a discography or a collection of albums, make sure albums are per folder.",
+            len(media_folders), difference)
 
         # While already failed, advice the user what he could try. We assume the
         # directory may contain separate CD's and maybe some extra's. The
         # structure may look like X albums at same depth, and (one or more)
         # extra folders with a higher depth.
-        extra_media_folders = [ media_folder[:min(path_depths)] for media_folder in media_folders if len(media_folder) > min(path_depths) ]
-        extra_media_folders = list(set([ os.path.join(*media_folder) for media_folder in extra_media_folders ]))
+        extra_media_folders = [media_folder[:min(path_depths)] for media_folder in media_folders if
+                               len(media_folder) > min(path_depths)]
+        extra_media_folders = list(
+            set([os.path.join(*media_folder) for media_folder in extra_media_folders]))
 
-        logger.info("Please look at the following folder(s), since they cause the depth difference: %s", extra_media_folders)
+        logger.info(
+            "Please look at the following folder(s), since they cause the depth difference: %s",
+            extra_media_folders)
         return
 
     # Convert back to paths and remove duplicates, which may be there after
     # correcting the paths
-    media_folders = list(set([ os.path.join(*media_folder) for media_folder in media_folders ]))
+    media_folders = list(set([os.path.join(*media_folder) for media_folder in media_folders]))
 
     # Don't return a result if the number of subfolders is one. In this case,
     # this algorithm will not improve processing and will likely interfere
@@ -307,12 +339,51 @@ def expand_subfolders(f):
     logger.debug("Expanded subfolders in folder: %s", media_folders)
     return media_folders
 
-def extract_data(s):
 
+def path_match_patterns(path, patterns):
+    """
+    Check if a path matches one or more patterns. The whole path will be
+    matched be matched against the patterns.
+    """
+
+    for pattern in patterns:
+        if fnmatch.fnmatch(path, pattern):
+            return True
+
+    # No match
+    return False
+
+
+def path_filter_patterns(paths, patterns, root=None):
+    """
+    Scan for ignored paths based on glob patterns. Note that the whole path
+    will be matched, therefore paths should only contain the relative paths.
+
+    The root is optional, and only used for producing meaningful debug info.
+    """
+
+    from headphones import logger
+
+    ignored = 0
+
+    for path in paths[:]:
+        if path_match_patterns(path, patterns):
+            logger.debug("Path ignored by pattern: %s",
+                         os.path.join(root or "", path))
+
+            ignored += 1
+            paths.remove(path)
+
+    # Return number of ignored paths
+    return ignored
+
+
+def extract_data(s):
     s = s.replace('_', ' ')
 
-    #headphones default format
-    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\[(?P<year>.*?)\]', re.VERBOSE)
+    # headphones default format
+    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s[\[\(](?P<year>.*?)[\]\)]',
+                         re.VERBOSE)
     match = pattern.match(s)
 
     if match:
@@ -321,16 +392,7 @@ def extract_data(s):
         year = match.group("year")
         return (name, album, year)
 
-    #newzbin default format
-    pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\((?P<year>\d+?\))', re.VERBOSE)
-    match = pattern.match(s)
-    if match:
-        name = match.group("name")
-        album = match.group("album")
-        year = match.group("year")
-        return (name, album, year)
-
-    #Gonna take a guess on this one - might be enough to search on mb
+    # Gonna take a guess on this one - might be enough to search on mb
     pat = re.compile(r"(?P<name>.*?)\s*-\s*(?P<album>[^\[(-]*)")
 
     match = pat.match(s)
@@ -343,6 +405,7 @@ def extract_data(s):
     else:
         return (None, None, None)
 
+
 def extract_metadata(f):
     """
     Scan all files in the given directory and decide on an artist, album and
@@ -392,9 +455,9 @@ def extract_metadata(f):
         return (None, None, None)
 
     # Count distinct values
-    artists = list(set([ x[0] for x in results ]))
-    albums = list(set([ x[1] for x in results ]))
-    years = list(set([ x[2] for x in results ]))
+    artists = list(set([x[0] for x in results]))
+    albums = list(set([x[1] for x in results]))
+    years = list(set([x[2] for x in results]))
 
     # Remove things such as CD2 from album names
     if len(albums) > 1:
@@ -406,7 +469,8 @@ def extract_metadata(f):
                 old_album = new_albums[index]
                 new_albums[index] = RE_CD_ALBUM.sub("", album).strip()
 
-                logger.debug("Stripped albumd number identifier: %s -> %s", old_album, new_albums[index])
+                logger.debug("Stripped albumd number identifier: %s -> %s", old_album,
+                             new_albums[index])
 
         # Remove duplicates
         new_albums = list(set(new_albums))
@@ -422,8 +486,8 @@ def extract_metadata(f):
 
     # (Lots of) different artists. Could be a featuring album, so test for this.
     if len(artists) > 1 and len(albums) == 1:
-        split_artists = [ RE_FEATURING.split(artist) for artist in artists ]
-        featurings = [ len(split_artist) - 1 for split_artist in split_artists ]
+        split_artists = [RE_FEATURING.split(x) for x in artists]
+        featurings = [len(split_artist) - 1 for split_artist in split_artists]
         logger.info("Album seem to feature %d different artists", sum(featurings))
 
         if sum(featurings) > 0:
@@ -436,14 +500,92 @@ def extract_metadata(f):
             return (artist, albums[0], years[0])
 
     # Not sure what to do here.
-    logger.info("Found %d artists, %d albums and %d years in metadata, so ignoring", len(artists), len(albums), len(years))
+    logger.info("Found %d artists, %d albums and %d years in metadata, so ignoring", len(artists),
+                len(albums), len(years))
     logger.debug("Artists: %s, Albums: %s, Years: %s", artists, albums, years)
 
     return (None, None, None)
 
+
+def get_downloaded_track_list(albumpath):
+    """
+     Return a list of audio files for the given directory.
+     """
+    downloaded_track_list = []
+
+    for root, dirs, files in os.walk(albumpath):
+        for _file in files:
+            extension = os.path.splitext(_file)[1].lower()[1:]
+            if extension in headphones.MEDIA_FORMATS:
+                downloaded_track_list.append(os.path.join(root, _file))
+
+    return downloaded_track_list
+
+
+def preserve_torrent_directory(albumpath):
+    """
+    Copy torrent directory to headphones-modified to keep files for seeding.
+    """
+    from headphones import logger
+    new_folder = os.path.join(albumpath,
+                              'headphones-modified'.encode(headphones.SYS_ENCODING, 'replace'))
+    logger.info(
+        "Copying files to 'headphones-modified' subfolder to preserve downloaded files for seeding")
+    try:
+        shutil.copytree(albumpath, new_folder)
+        return new_folder
+    except Exception as e:
+        logger.warn("Cannot copy/move files to temp folder: " + \
+                    new_folder.decode(headphones.SYS_ENCODING, 'replace') + \
+                    ". Not continuing. Error: " + str(e))
+        return None
+
+
+def cue_split(albumpath):
+    """
+     Attempts to check and split audio files by a cue for the given directory.
+     """
+    # Walk directory and scan all media files
+    count = 0
+    cue_count = 0
+    cue_dirs = []
+
+    for root, dirs, files in os.walk(albumpath):
+        for _file in files:
+            extension = os.path.splitext(_file)[1].lower()[1:]
+            if extension in headphones.MEDIA_FORMATS:
+                count += 1
+            elif extension == 'cue':
+                cue_count += 1
+                if root not in cue_dirs:
+                    cue_dirs.append(root)
+
+    # Split cue
+    if cue_count and cue_count >= count and cue_dirs:
+
+        from headphones import logger, cuesplit
+        logger.info("Attempting to split audio files by cue")
+
+        cwd = os.getcwd()
+        for cue_dir in cue_dirs:
+            try:
+                cuesplit.split(cue_dir)
+            except Exception as e:
+                os.chdir(cwd)
+                logger.warn("Cue not split: " + str(e))
+                return False
+
+        os.chdir(cwd)
+        return True
+
+    return False
+
+
 def extract_logline(s):
     # Default log format
-    pattern = re.compile(r'(?P<timestamp>.*?)\s\-\s(?P<level>.*?)\s*\:\:\s(?P<thread>.*?)\s\:\s(?P<message>.*)', re.VERBOSE)
+    pattern = re.compile(
+        r'(?P<timestamp>.*?)\s\-\s(?P<level>.*?)\s*\:\:\s(?P<thread>.*?)\s\:\s(?P<message>.*)',
+        re.VERBOSE)
     match = pattern.match(s)
     if match:
         timestamp = match.group("timestamp")
@@ -454,14 +596,11 @@ def extract_logline(s):
     else:
         return None
 
+
 def extract_song_data(s):
+    from headphones import logger
 
-    #headphones default format
-    music_dir = headphones.MUSIC_DIR
-    folder_format = headphones.FOLDER_FORMAT
-    file_format = headphones.FILE_FORMAT
-
-    full_format = os.path.join(headphones.MUSIC_DIR)
+    # headphones default format
     pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\[(?P<year>.*?)\]', re.VERBOSE)
     match = pattern.match(s)
 
@@ -473,7 +612,7 @@ def extract_song_data(s):
     else:
         logger.info("Couldn't parse %s into a valid default format", s)
 
-    #newzbin default format
+    # newzbin default format
     pattern = re.compile(r'(?P<name>.*?)\s\-\s(?P<album>.*?)\s\((?P<year>\d+?\))', re.VERBOSE)
     match = pattern.match(s)
     if match:
@@ -485,8 +624,8 @@ def extract_song_data(s):
         logger.info("Couldn't parse %s into a valid Newbin format", s)
         return (name, album, year)
 
-def smartMove(src, dest, delete=True):
 
+def smartMove(src, dest, delete=True):
     from headphones import logger
 
     source_dir = os.path.dirname(src)
@@ -506,8 +645,9 @@ def smartMove(src, dest, delete=True):
                 try:
                     os.rename(src, os.path.join(source_dir, newfile))
                     filename = newfile
-                except Exception, e:
-                    logger.warn('Error renaming %s: %s', src.decode(headphones.SYS_ENCODING, 'replace'), e)
+                except Exception as e:
+                    logger.warn('Error renaming %s: %s',
+                                src.decode(headphones.SYS_ENCODING, 'replace'), e)
                 break
 
     try:
@@ -516,41 +656,85 @@ def smartMove(src, dest, delete=True):
         else:
             shutil.copy(os.path.join(source_dir, filename), os.path.join(dest, filename))
             return True
-    except Exception, e:
-        logger.warn('Error moving file %s: %s', filename.decode(headphones.SYS_ENCODING, 'replace'), e)
+    except Exception as e:
+        logger.warn('Error moving file %s: %s', filename.decode(headphones.SYS_ENCODING, 'replace'),
+                    e)
+
+
+def walk_directory(basedir, followlinks=True):
+    """
+    Enhanced version of 'os.walk' where symlink directores are traversed, but
+    with care. In case a folder is already processed, don't traverse it again.
+    """
+
+    import logger
+
+    # Add the base path, because symlinks poiting to the basedir should not be
+    # traversed again.
+    traversed = [os.path.abspath(basedir)]
+
+    def _inner(root, directories, files):
+        for directory in directories:
+            path = os.path.join(root, directory)
+
+            if followlinks and os.path.islink(path):
+                real_path = os.path.abspath(os.readlink(path))
+
+                if real_path in traversed:
+                    logger.debug("Skipping '%s' since it is a symlink to " \
+                                 "'%s', which is already visited.", path, real_path)
+                else:
+                    traversed.append(real_path)
+
+                    for args in os.walk(real_path):
+                        for result in _inner(*args):
+                            yield result
+
+        # Pass on actual result
+        yield root, directories, files
+
+    # Start traversing
+    for args in os.walk(basedir):
+        for result in _inner(*args):
+            yield result
+
 
 #########################
-#Sab renaming functions #
+# Sab renaming functions #
 #########################
 
 # TODO: Grab config values from sab to know when these options are checked. For now we'll just iterate through all combinations
 
+
 def sab_replace_dots(name):
-    return name.replace('.',' ')
+    return name.replace('.', ' ')
+
+
 def sab_replace_spaces(name):
-    return name.replace(' ','_')
+    return name.replace(' ', '_')
+
 
 def sab_sanitize_foldername(name):
     """ Return foldername with dodgy chars converted to safe ones
         Remove any leading and trailing dot and space characters
     """
     CH_ILLEGAL = r'\/<>?*|"'
-    CH_LEGAL   = r'++{}!@#`'
+    CH_LEGAL = r'++{}!@#`'
 
     FL_ILLEGAL = CH_ILLEGAL + ':\x92"'
-    FL_LEGAL   = CH_LEGAL +   "-''"
+    FL_LEGAL = CH_LEGAL + "-''"
 
     uFL_ILLEGAL = FL_ILLEGAL.decode('latin-1')
-    uFL_LEGAL   = FL_LEGAL.decode('latin-1')
+    uFL_LEGAL = FL_LEGAL.decode('latin-1')
 
     if not name:
         return name
     if isinstance(name, unicode):
         illegal = uFL_ILLEGAL
-        legal   = uFL_LEGAL
+        legal = uFL_LEGAL
     else:
         illegal = FL_ILLEGAL
-        legal   = FL_LEGAL
+        legal = FL_LEGAL
 
     lst = []
     for ch in name.strip():
@@ -565,48 +749,51 @@ def sab_sanitize_foldername(name):
     if not name:
         name = 'unknown'
 
-    #maxlen = cfg.folder_max_length()
-    #if len(name) > maxlen:
+    # maxlen = cfg.folder_max_length()
+    # if len(name) > maxlen:
     #    name = name[:maxlen]
 
     return name
 
+
 def split_string(mystring, splitvar=','):
     mylist = []
     for each_word in mystring.split(splitvar):
         mylist.append(each_word.strip())
     return mylist
 
+
 def create_https_certificates(ssl_cert, ssl_key):
     """
-    Stolen from SickBeard (http://github.com/midgetspy/Sick-Beard):
-    Create self-signed HTTPS certificares and store in paths 'ssl_cert' and 'ssl_key'
+    Create a pair of self-signed HTTPS certificares and store in them in
+    'ssl_cert' and 'ssl_key'. Method assumes pyOpenSSL is installed.
+
+    This code is stolen from SickBeard (http://github.com/midgetspy/Sick-Beard).
     """
+
     from headphones import logger
 
-    try:
-        from OpenSSL import crypto
-        from lib.certgen import createKeyPair, createCertRequest, createCertificate, TYPE_RSA, serial
-    except:
-        logger.warn("pyOpenSSL module missing, please install to enable HTTPS")
-        return False
+    from OpenSSL import crypto
+    from certgen import createKeyPair, createCertRequest, createCertificate, \
+        TYPE_RSA, serial
 
     # Create the CA Certificate
-    cakey = createKeyPair(TYPE_RSA, 1024)
-    careq = createCertRequest(cakey, CN='Certificate Authority')
-    cacert = createCertificate(careq, (careq, cakey), serial, (0, 60*60*24*365*10)) # ten years
+    cakey = createKeyPair(TYPE_RSA, 2048)
+    careq = createCertRequest(cakey, CN="Certificate Authority")
+    cacert = createCertificate(careq, (careq, cakey), serial,
+                               (0, 60 * 60 * 24 * 365 * 10))  # ten years
 
-    cname = 'Headphones'
-    pkey = createKeyPair(TYPE_RSA, 1024)
-    req = createCertRequest(pkey, CN=cname)
-    cert = createCertificate(req, (cacert, cakey), serial, (0, 60*60*24*365*10)) # ten years
+    pkey = createKeyPair(TYPE_RSA, 2048)
+    req = createCertRequest(pkey, CN="Headphones")
+    cert = createCertificate(req, (cacert, cakey), serial,
+                             (0, 60 * 60 * 24 * 365 * 10))  # ten years
 
     # Save the key and certificate to disk
     try:
-        with open(ssl_key, 'w') as f:
-            f.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, pkey))
-        with open(ssl_cert, 'w') as f:
-            f.write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
+        with open(ssl_key, "w") as fp:
+            fp.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, pkey))
+        with open(ssl_cert, "w") as fp:
+            fp.write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
     except IOError as e:
         logger.error("Error creating SSL key and certificate: %s", e)
         return False

headphones/importer.py

@@ -13,46 +13,44 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger, helpers, db, mb, lastfm
-
-from beets.mediafile import MediaFile
-
-import os
 import time
-import threading
+
+from headphones import logger, helpers, db, mb, lastfm, metacritic
+from beets.mediafile import MediaFile
 import headphones
 
 blacklisted_special_artist_names = ['[anonymous]', '[data]', '[no artist]',
-    '[traditional]','[unknown]','Various Artists']
+                                    '[traditional]', '[unknown]', 'Various Artists']
 blacklisted_special_artists = ['f731ccc4-e22a-43af-a747-64213329e088',
-    '33cf029c-63b0-41a0-9855-be2a3665fb3b',
-    '314e1c25-dde7-4e4d-b2f4-0a7b9f7c56dc',
-    'eec63d3c-3b81-4ad4-b1e4-7c147d4d2b61',
-    '9be7f096-97ec-4615-8957-8d40b5dcbc41',
-    '125ec42a-7229-4250-afc5-e057484327fe',
-    '89ad4ac3-39f7-470e-963a-56509c546377']
+                               '33cf029c-63b0-41a0-9855-be2a3665fb3b',
+                               '314e1c25-dde7-4e4d-b2f4-0a7b9f7c56dc',
+                               'eec63d3c-3b81-4ad4-b1e4-7c147d4d2b61',
+                               '9be7f096-97ec-4615-8957-8d40b5dcbc41',
+                               '125ec42a-7229-4250-afc5-e057484327fe',
+                               '89ad4ac3-39f7-470e-963a-56509c546377']
+
 
 def is_exists(artistid):
     myDB = db.DBConnection()
 
     # See if the artist is already in the database
-    artistlist = myDB.select('SELECT ArtistID, ArtistName from artists WHERE ArtistID=?', [artistid])
+    artistlist = myDB.select('SELECT ArtistID, ArtistName from artists WHERE ArtistID=?',
+                             [artistid])
 
     if any(artistid in x for x in artistlist):
-        logger.info(artistlist[0][1] + u" is already in the database. Updating 'have tracks', but not artist information")
+        logger.info(artistlist[0][
+                        1] + u" is already in the database. Updating 'have tracks', but not artist information")
         return True
     else:
         return False
 
 
 def artistlist_to_mbids(artistlist, forced=False):
-
     for artist in artistlist:
 
-        if not artist and not (artist == ' '):
+        if not artist and artist != ' ':
             continue
 
-
         # If adding artists through Manage New Artists, they're coming through as non-unicode (utf-8?)
         # and screwing everything up
         if not isinstance(artist, unicode):
@@ -79,9 +77,12 @@ def artistlist_to_mbids(artistlist, forced=False):
         myDB = db.DBConnection()
 
         if not forced:
-            bl_artist = myDB.action('SELECT * FROM blacklist WHERE ArtistID=?', [artistid]).fetchone()
+            bl_artist = myDB.action('SELECT * FROM blacklist WHERE ArtistID=?',
+                                    [artistid]).fetchone()
             if bl_artist or artistid in blacklisted_special_artists:
-                logger.info("Artist ID for '%s' is either blacklisted or Various Artists. To add artist, you must do it manually (Artist ID: %s)" % (artist, artistid))
+                logger.info(
+                    "Artist ID for '%s' is either blacklisted or Various Artists. To add artist, you must do it manually (Artist ID: %s)" % (
+                    artist, artistid))
                 continue
 
         # Add to database if it doesn't exist
@@ -90,7 +91,9 @@ def artistlist_to_mbids(artistlist, forced=False):
 
         # Just update the tracks if it does
         else:
-            havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=?', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist]))
+            havetracks = len(
+                myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=?', [artistid])) + len(
+                myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist]))
             myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistID=?', [havetracks, artistid])
 
         # Delete it from the New Artists if the request came from there
@@ -105,14 +108,15 @@ def artistlist_to_mbids(artistlist, forced=False):
     except Exception as e:
         logger.warn('Failed to update arist information from Last.fm: %s' % e)
 
+
 def addArtistIDListToDB(artistidlist):
     # Used to add a list of artist IDs to the database in a single thread
     logger.debug("Importer: Adding artist ids %s" % artistidlist)
     for artistid in artistidlist:
         addArtisttoDB(artistid)
 
-def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
+def addArtisttoDB(artistid, extrasonly=False, forcefull=False, type="artist"):
     # Putting this here to get around the circular import. We're using this to update thumbnails for artist/albums
     from headphones import cache
 
@@ -131,28 +135,35 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
     # We need the current minimal info in the database instantly
     # so we don't throw a 500 error when we redirect to the artistPage
-    controlValueDict = {"ArtistID":     artistid}
+    controlValueDict = {"ArtistID": artistid}
 
     # Don't replace a known artist name with an "Artist ID" placeholder
     dbartist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [artistid]).fetchone()
 
     # Only modify the Include Extras stuff if it's a new artist. We need it early so we know what to fetch
     if not dbartist:
-        newValueDict = {"ArtistName":   "Artist ID: %s" % (artistid),
-                        "Status":       "Loading",
-                        "IncludeExtras": headphones.INCLUDE_EXTRAS,
-                        "Extras":        headphones.EXTRAS }
+        newValueDict = {"ArtistName": "Artist ID: %s" % (artistid),
+                        "Status": "Loading",
+                        "IncludeExtras": headphones.CONFIG.INCLUDE_EXTRAS,
+                        "Extras": headphones.CONFIG.EXTRAS}
+        if type == "series":
+            newValueDict['Type'] = "series"
     else:
-        newValueDict = {"Status":   "Loading"}
+        newValueDict = {"Status": "Loading"}
+        if dbartist["Type"] == "series":
+            type = "series"
 
     myDB.upsert("artists", newValueDict, controlValueDict)
 
-    artist = mb.getArtist(artistid, extrasonly)
+    if type == "series":
+        artist = mb.getSeries(artistid)
+    else:
+        artist = mb.getArtist(artistid, extrasonly)
 
     if artist and artist.get('artist_name') in blacklisted_special_artist_names:
         logger.warn('Cannot import blocked special purpose artist: %s' % artist.get('artist_name'))
         myDB.action('DELETE from artists WHERE ArtistID=?', [artistid])
-        #in case it's already in the db
+        # in case it's already in the db
         myDB.action('DELETE from albums WHERE ArtistID=?', [artistid])
         myDB.action('DELETE from tracks WHERE ArtistID=?', [artistid])
         return
@@ -160,10 +171,10 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
     if not artist:
         logger.warn("Error fetching artist info. ID: " + artistid)
         if dbartist is None:
-            newValueDict = {"ArtistName":   "Fetch failed, try refreshing. (%s)" % (artistid),
-                    "Status":   "Active"}
+            newValueDict = {"ArtistName": "Fetch failed, try refreshing. (%s)" % (artistid),
+                            "Status": "Active"}
         else:
-            newValueDict = {"Status":   "Active"}
+            newValueDict = {"Status": "Active"}
         myDB.upsert("artists", newValueDict, controlValueDict)
         return
 
@@ -172,20 +183,20 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
     else:
         sortname = artist['artist_name']
 
-
     logger.info(u"Now adding/updating: " + artist['artist_name'])
-    controlValueDict = {"ArtistID":     artistid}
-    newValueDict = {"ArtistName":       artist['artist_name'],
-                    "ArtistSortName":   sortname,
-                    "DateAdded":        helpers.today(),
-                    "Status":           "Loading"}
+    controlValueDict = {"ArtistID": artistid}
+    newValueDict = {"ArtistName": artist['artist_name'],
+                    "ArtistSortName": sortname,
+                    "DateAdded": helpers.today(),
+                    "Status": "Loading"}
 
     myDB.upsert("artists", newValueDict, controlValueDict)
 
     # See if we need to grab extras. Artist specific extras take precedence
     # over global option. Global options are set when adding a new artist
     try:
-        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?', [artistid]).fetchone()
+        db_artist = myDB.action('SELECT IncludeExtras, Extras from artists WHERE ArtistID=?',
+                                [artistid]).fetchone()
         includeExtras = db_artist['IncludeExtras']
     except IndexError:
         includeExtras = False
@@ -200,9 +211,12 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
         for groups in artist['releasegroups']:
             group_list.append(groups['id'])
         if not extrasonly:
-            remove_missing_groups_from_albums = myDB.select("SELECT AlbumID FROM albums WHERE ArtistID=?", [artistid])
+            remove_missing_groups_from_albums = myDB.select(
+                "SELECT AlbumID FROM albums WHERE ArtistID=?", [artistid])
         else:
-            remove_missing_groups_from_albums = myDB.select('SELECT AlbumID FROM albums WHERE ArtistID=? AND Status="Skipped" AND Type!="Album"', [artistid])
+            remove_missing_groups_from_albums = myDB.select(
+                'SELECT AlbumID FROM albums WHERE ArtistID=? AND Status="Skipped" AND Type!="Album"',
+                [artistid])
         for items in remove_missing_groups_from_albums:
             if items['AlbumID'] not in group_list:
                 # Remove all from albums/tracks that aren't in release groups
@@ -211,12 +225,16 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                 myDB.action("DELETE FROM tracks WHERE AlbumID=?", [items['AlbumID']])
                 myDB.action("DELETE FROM alltracks WHERE AlbumID=?", [items['AlbumID']])
                 myDB.action('DELETE from releases WHERE ReleaseGroupID=?', [items['AlbumID']])
-                logger.info("[%s] Removing all references to release group %s to reflect MusicBrainz refresh" % (artist['artist_name'], items['AlbumID']))
+                logger.info(
+                    "[%s] Removing all references to release group %s to reflect MusicBrainz refresh" % (
+                    artist['artist_name'], items['AlbumID']))
                 if not extrasonly:
                     force_repackage = 1
     else:
         if not extrasonly:
-            logger.info("[%s] There was either an error pulling data from MusicBrainz or there might not be any releases for this category" % artist['artist_name'])
+            logger.info(
+                "[%s] There was either an error pulling data from MusicBrainz or there might not be any releases for this category" %
+                artist['artist_name'])
 
     # Then search for releases within releasegroups, if releases don't exist, then remove from allalbums/alltracks
     album_searches = []
@@ -226,8 +244,8 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
         today = helpers.today()
         rgid = rg['id']
         skip_log = 0
-        #Make a user configurable variable to skip update of albums with release dates older than this date (in days)
-        pause_delta = headphones.MB_IGNORE_AGE
+        # Make a user configurable variable to skip update of albums with release dates older than this date (in days)
+        pause_delta = headphones.CONFIG.MB_IGNORE_AGE
 
         rg_exists = myDB.action("SELECT * from albums WHERE AlbumID=?", [rg['id']]).fetchone()
 
@@ -240,40 +258,45 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
                 check_release_date = None
                 new_release_group = True
 
-
             if new_release_group:
-                logger.info("[%s] Now adding: %s (New Release Group)" % (artist['artist_name'], rg['title']))
-                new_releases = mb.get_new_releases(rgid,includeExtras)
+                logger.info("[%s] Now adding: %s (New Release Group)" % (
+                artist['artist_name'], rg['title']))
+                new_releases = mb.get_new_releases(rgid, includeExtras)
 
             else:
                 if check_release_date is None or check_release_date == u"None":
-                    logger.info("[%s] Now updating: %s (No Release Date)" % (artist['artist_name'], rg['title']))
-                    new_releases = mb.get_new_releases(rgid,includeExtras,True)
+                    logger.info("[%s] Now updating: %s (No Release Date)" % (
+                    artist['artist_name'], rg['title']))
+                    new_releases = mb.get_new_releases(rgid, includeExtras, True)
                 else:
                     if len(check_release_date) == 10:
                         release_date = check_release_date
                     elif len(check_release_date) == 7:
-                        release_date = check_release_date+"-31"
+                        release_date = check_release_date + "-31"
                     elif len(check_release_date) == 4:
-                        release_date = check_release_date+"-12-31"
+                        release_date = check_release_date + "-12-31"
                     else:
                         release_date = today
                     if helpers.get_age(today) - helpers.get_age(release_date) < pause_delta:
-                        logger.info("[%s] Now updating: %s (Release Date <%s Days)", artist['artist_name'], rg['title'], pause_delta)
-                        new_releases = mb.get_new_releases(rgid,includeExtras,True)
+                        logger.info("[%s] Now updating: %s (Release Date <%s Days)",
+                                    artist['artist_name'], rg['title'], pause_delta)
+                        new_releases = mb.get_new_releases(rgid, includeExtras, True)
                     else:
-                        logger.info("[%s] Skipping: %s (Release Date >%s Days)", artist['artist_name'], rg['title'], pause_delta)
+                        logger.info("[%s] Skipping: %s (Release Date >%s Days)",
+                                    artist['artist_name'], rg['title'], pause_delta)
                         skip_log = 1
                         new_releases = 0
 
             if force_repackage == 1:
                 new_releases = -1
-                logger.info('[%s] Forcing repackage of %s (Release Group Removed)', artist['artist_name'], al_title)
+                logger.info('[%s] Forcing repackage of %s (Release Group Removed)',
+                            artist['artist_name'], al_title)
             else:
                 new_releases = new_releases
         else:
-            logger.info("[%s] Now adding/updating: %s (Comprehensive Force)", artist['artist_name'], rg['title'])
-            new_releases = mb.get_new_releases(rgid,includeExtras,forcefull)
+            logger.info("[%s] Now adding/updating: %s (Comprehensive Force)", artist['artist_name'],
+                        rg['title'])
+            new_releases = mb.get_new_releases(rgid, includeExtras, forcefull)
 
         if new_releases != 0:
             # Dump existing hybrid release since we're repackaging/replacing it
@@ -286,33 +309,36 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
             # This will be used later to build a hybrid release
             fullreleaselist = []
             # Search for releases within a release group
-            find_hybrid_releases = myDB.action("SELECT * from allalbums WHERE AlbumID=?", [rg['id']])
+            find_hybrid_releases = myDB.action("SELECT * from allalbums WHERE AlbumID=?",
+                                               [rg['id']])
 
             # Build the dictionary for the fullreleaselist
             for items in find_hybrid_releases:
-                if items['ReleaseID'] != rg['id']: #don't include hybrid information, since that's what we're replacing
+                if items['ReleaseID'] != rg[
+                    'id']:  # don't include hybrid information, since that's what we're replacing
                     hybrid_release_id = items['ReleaseID']
-                    newValueDict = {"ArtistID":         items['ArtistID'],
-                        "ArtistName":       items['ArtistName'],
-                        "AlbumTitle":       items['AlbumTitle'],
-                        "AlbumID":          items['AlbumID'],
-                        "AlbumASIN":        items['AlbumASIN'],
-                        "ReleaseDate":      items['ReleaseDate'],
-                        "Type":             items['Type'],
-                        "ReleaseCountry":   items['ReleaseCountry'],
-                        "ReleaseFormat":    items['ReleaseFormat']
-                    }
-                    find_hybrid_tracks = myDB.action("SELECT * from alltracks WHERE ReleaseID=?", [hybrid_release_id])
+                    newValueDict = {"ArtistID": items['ArtistID'],
+                                    "ArtistName": items['ArtistName'],
+                                    "AlbumTitle": items['AlbumTitle'],
+                                    "AlbumID": items['AlbumID'],
+                                    "AlbumASIN": items['AlbumASIN'],
+                                    "ReleaseDate": items['ReleaseDate'],
+                                    "Type": items['Type'],
+                                    "ReleaseCountry": items['ReleaseCountry'],
+                                    "ReleaseFormat": items['ReleaseFormat']
+                                    }
+                    find_hybrid_tracks = myDB.action("SELECT * from alltracks WHERE ReleaseID=?",
+                                                     [hybrid_release_id])
                     totalTracks = 1
                     hybrid_track_array = []
                     for hybrid_tracks in find_hybrid_tracks:
                         hybrid_track_array.append({
-                            'number':        hybrid_tracks['TrackNumber'],
-                            'title':         hybrid_tracks['TrackTitle'],
-                            'id':            hybrid_tracks['TrackID'],
-                            #'url':           hybrid_tracks['TrackURL'],
-                            'duration':      hybrid_tracks['TrackDuration']
-                            })
+                            'number': hybrid_tracks['TrackNumber'],
+                            'title': hybrid_tracks['TrackTitle'],
+                            'id': hybrid_tracks['TrackID'],
+                            # 'url':           hybrid_tracks['TrackURL'],
+                            'duration': hybrid_tracks['TrackDuration']
+                        })
                         totalTracks += 1
                     newValueDict['ReleaseID'] = hybrid_release_id
                     newValueDict['Tracks'] = hybrid_track_array
@@ -322,63 +348,70 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
             # This may end up being called with an empty fullreleaselist
             try:
                 hybridrelease = getHybridRelease(fullreleaselist)
-                logger.info('[%s] Packaging %s releases into hybrid title' % (artist['artist_name'], rg['title']))
+                logger.info('[%s] Packaging %s releases into hybrid title' % (
+                artist['artist_name'], rg['title']))
             except Exception as e:
                 errors = True
-                logger.warn('[%s] Unable to get hybrid release information for %s: %s' % (artist['artist_name'],rg['title'],e))
+                logger.warn('[%s] Unable to get hybrid release information for %s: %s' % (
+                artist['artist_name'], rg['title'], e))
                 continue
 
             # Use the ReleaseGroupID as the ReleaseID for the hybrid release to differentiate it
             # We can then use the condition WHERE ReleaseID == ReleaseGroupID to select it
             # The hybrid won't have a country or a format
-            controlValueDict = {"ReleaseID":  rg['id']}
+            controlValueDict = {"ReleaseID": rg['id']}
 
-            newValueDict = {"ArtistID":         artistid,
-                            "ArtistName":       artist['artist_name'],
-                            "AlbumTitle":       rg['title'],
-                            "AlbumID":          rg['id'],
-                            "AlbumASIN":        hybridrelease['AlbumASIN'],
-                            "ReleaseDate":      hybridrelease['ReleaseDate'],
-                            "Type":             rg['type']
-                        }
+            newValueDict = {"ArtistID": artistid,
+                            "ArtistName": artist['artist_name'],
+                            "AlbumTitle": rg['title'],
+                            "AlbumID": rg['id'],
+                            "AlbumASIN": hybridrelease['AlbumASIN'],
+                            "ReleaseDate": hybridrelease['ReleaseDate'],
+                            "Type": rg['type']
+                            }
 
             myDB.upsert("allalbums", newValueDict, controlValueDict)
 
             for track in hybridrelease['Tracks']:
 
-                cleanname = helpers.cleanName(artist['artist_name'] + ' ' + rg['title'] + ' ' + track['title'])
+                cleanname = helpers.cleanName(
+                    artist['artist_name'] + ' ' + rg['title'] + ' ' + track['title'])
 
-                controlValueDict = {"TrackID":      track['id'],
-                                    "ReleaseID":    rg['id']}
+                controlValueDict = {"TrackID": track['id'],
+                                    "ReleaseID": rg['id']}
 
-                newValueDict = {"ArtistID":         artistid,
-                                "ArtistName":       artist['artist_name'],
-                                "AlbumTitle":       rg['title'],
-                                "AlbumASIN":        hybridrelease['AlbumASIN'],
-                                "AlbumID":          rg['id'],
-                                "TrackTitle":       track['title'],
-                                "TrackDuration":    track['duration'],
-                                "TrackNumber":      track['number'],
-                                "CleanName":        cleanname
-                            }
+                newValueDict = {"ArtistID": artistid,
+                                "ArtistName": artist['artist_name'],
+                                "AlbumTitle": rg['title'],
+                                "AlbumASIN": hybridrelease['AlbumASIN'],
+                                "AlbumID": rg['id'],
+                                "TrackTitle": track['title'],
+                                "TrackDuration": track['duration'],
+                                "TrackNumber": track['number'],
+                                "CleanName": cleanname
+                                }
 
-                match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?', [cleanname]).fetchone()
+                match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?',
+                                    [cleanname]).fetchone()
 
                 if not match:
-                    match = myDB.action('SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [artist['artist_name'], rg['title'], track['title']]).fetchone()
-                #if not match:
-                    #match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
+                    match = myDB.action(
+                        'SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                        [artist['artist_name'], rg['title'], track['title']]).fetchone()
+                    # if not match:
+                    # match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
                 if match:
                     newValueDict['Location'] = match['Location']
                     newValueDict['BitRate'] = match['BitRate']
                     newValueDict['Format'] = match['Format']
-                    #myDB.action('UPDATE have SET Matched="True" WHERE Location=?', [match['Location']])
-                    myDB.action('UPDATE have SET Matched=? WHERE Location=?', (rg['id'], match['Location']))
+                    # myDB.action('UPDATE have SET Matched="True" WHERE Location=?', [match['Location']])
+                    myDB.action('UPDATE have SET Matched=? WHERE Location=?',
+                                (rg['id'], match['Location']))
 
                 myDB.upsert("alltracks", newValueDict, controlValueDict)
 
             # Delete matched tracks from the have table
-            #myDB.action('DELETE from have WHERE Matched="True"')
+            # myDB.action('DELETE from have WHERE Matched="True"')
 
             # If there's no release in the main albums tables, add the default (hybrid)
             # If there is a release, check the ReleaseID against the AlbumID to see if they differ (user updated)
@@ -392,42 +425,45 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
             album = myDB.action('SELECT * from allalbums WHERE ReleaseID=?', [releaseid]).fetchone()
 
-            controlValueDict = {"AlbumID":  rg['id']}
+            controlValueDict = {"AlbumID": rg['id']}
 
-            newValueDict = {"ArtistID":         album['ArtistID'],
-                            "ArtistName":       album['ArtistName'],
-                            "AlbumTitle":       album['AlbumTitle'],
-                            "ReleaseID":        album['ReleaseID'],
-                            "AlbumASIN":        album['AlbumASIN'],
-                            "ReleaseDate":      album['ReleaseDate'],
-                            "Type":             album['Type'],
-                            "ReleaseCountry":   album['ReleaseCountry'],
-                            "ReleaseFormat":    album['ReleaseFormat']
-                        }
+            newValueDict = {"ArtistID": album['ArtistID'],
+                            "ArtistName": album['ArtistName'],
+                            "AlbumTitle": album['AlbumTitle'],
+                            "ReleaseID": album['ReleaseID'],
+                            "AlbumASIN": album['AlbumASIN'],
+                            "ReleaseDate": album['ReleaseDate'],
+                            "Type": album['Type'],
+                            "ReleaseCountry": album['ReleaseCountry'],
+                            "ReleaseFormat": album['ReleaseFormat']
+                            }
 
             if rg_exists:
                 newValueDict['DateAdded'] = rg_exists['DateAdded']
-                newValueDict['Status']    = rg_exists['Status']
+                newValueDict['Status'] = rg_exists['Status']
 
             else:
                 today = helpers.today()
 
                 newValueDict['DateAdded'] = today
 
-                if headphones.AUTOWANT_ALL:
+                if headphones.CONFIG.AUTOWANT_ALL:
                     newValueDict['Status'] = "Wanted"
-                elif album['ReleaseDate'] > today and headphones.AUTOWANT_UPCOMING:
+                elif album['ReleaseDate'] > today and headphones.CONFIG.AUTOWANT_UPCOMING:
                     newValueDict['Status'] = "Wanted"
                 # Sometimes "new" albums are added to musicbrainz after their release date, so let's try to catch these
                 # The first test just makes sure we have year-month-day
-                elif helpers.get_age(album['ReleaseDate']) and helpers.get_age(today) - helpers.get_age(album['ReleaseDate']) < 21 and headphones.AUTOWANT_UPCOMING:
+                elif helpers.get_age(album['ReleaseDate']) and helpers.get_age(
+                        today) - helpers.get_age(
+                        album['ReleaseDate']) < 21 and headphones.CONFIG.AUTOWANT_UPCOMING:
                     newValueDict['Status'] = "Wanted"
                 else:
                     newValueDict['Status'] = "Skipped"
 
             myDB.upsert("albums", newValueDict, controlValueDict)
 
-            tracks = myDB.action('SELECT * from alltracks WHERE ReleaseID=?', [releaseid]).fetchall()
+            tracks = myDB.action('SELECT * from alltracks WHERE ReleaseID=?',
+                                 [releaseid]).fetchall()
 
             # This is used to see how many tracks you have from an album - to
             # mark it as downloaded. Default is 80%, can be set in config as
@@ -436,53 +472,62 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
 
             if total_track_count == 0:
                 logger.warning("Total track count is zero for Release ID " +
-                    "'%s', skipping.", releaseid)
+                               "'%s', skipping.", releaseid)
                 continue
 
             for track in tracks:
-                controlValueDict = {"TrackID":  track['TrackID'],
-                                    "AlbumID":  rg['id']}
+                controlValueDict = {"TrackID": track['TrackID'],
+                                    "AlbumID": rg['id']}
 
-                newValueDict = {"ArtistID":     track['ArtistID'],
-                            "ArtistName":       track['ArtistName'],
-                            "AlbumTitle":       track['AlbumTitle'],
-                            "AlbumASIN":        track['AlbumASIN'],
-                            "ReleaseID":        track['ReleaseID'],
-                            "TrackTitle":       track['TrackTitle'],
-                            "TrackDuration":    track['TrackDuration'],
-                            "TrackNumber":      track['TrackNumber'],
-                            "CleanName":        track['CleanName'],
-                            "Location":         track['Location'],
-                            "Format":           track['Format'],
-                            "BitRate":          track['BitRate']
-                            }
+                newValueDict = {"ArtistID": track['ArtistID'],
+                                "ArtistName": track['ArtistName'],
+                                "AlbumTitle": track['AlbumTitle'],
+                                "AlbumASIN": track['AlbumASIN'],
+                                "ReleaseID": track['ReleaseID'],
+                                "TrackTitle": track['TrackTitle'],
+                                "TrackDuration": track['TrackDuration'],
+                                "TrackNumber": track['TrackNumber'],
+                                "CleanName": track['CleanName'],
+                                "Location": track['Location'],
+                                "Format": track['Format'],
+                                "BitRate": track['BitRate']
+                                }
 
                 myDB.upsert("tracks", newValueDict, controlValueDict)
 
             # Mark albums as downloaded if they have at least 80% (by default, configurable) of the album
-            have_track_count = len(myDB.select('SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL', [rg['id']]))
+            have_track_count = len(
+                myDB.select('SELECT * from tracks WHERE AlbumID=? AND Location IS NOT NULL',
+                            [rg['id']]))
             marked_as_downloaded = False
 
             if rg_exists:
-                if rg_exists['Status'] == 'Skipped' and ((have_track_count/float(total_track_count)) >= (headphones.ALBUM_COMPLETION_PCT/100.0)):
-                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
+                if rg_exists['Status'] == 'Skipped' and (
+                    (have_track_count / float(total_track_count)) >= (
+                    headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0)):
+                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?',
+                                ['Downloaded', rg['id']])
                     marked_as_downloaded = True
             else:
-                if ((have_track_count/float(total_track_count)) >= (headphones.ALBUM_COMPLETION_PCT/100.0)):
-                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?', ['Downloaded', rg['id']])
+                if (have_track_count / float(total_track_count)) >= (
+                    headphones.CONFIG.ALBUM_COMPLETION_PCT / 100.0):
+                    myDB.action('UPDATE albums SET Status=? WHERE AlbumID=?',
+                                ['Downloaded', rg['id']])
                     marked_as_downloaded = True
 
-            logger.info(u"[%s] Seeing if we need album art for %s" % (artist['artist_name'], rg['title']))
+            logger.info(
+                u"[%s] Seeing if we need album art for %s" % (artist['artist_name'], rg['title']))
             cache.getThumb(AlbumID=rg['id'])
 
             # Start a search for the album if it's new, hasn't been marked as
             # downloaded and autowant_all is selected. This search is deferred,
             # in case the search failes and the rest of the import will halt.
-            if not rg_exists and not marked_as_downloaded and headphones.AUTOWANT_ALL:
+            if not rg_exists and not marked_as_downloaded and headphones.CONFIG.AUTOWANT_ALL:
                 album_searches.append(rg['id'])
         else:
             if skip_log == 0:
-                logger.info(u"[%s] No new releases, so no changes made to %s" % (artist['artist_name'], rg['title']))
+                logger.info(u"[%s] No new releases, so no changes made to %s" % (
+                artist['artist_name'], rg['title']))
 
     time.sleep(3)
     finalize_update(artistid, artist['artist_name'], errors)
@@ -490,8 +535,13 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
     logger.info(u"Seeing if we need album art for: %s" % artist['artist_name'])
     cache.getThumb(ArtistID=artistid)
 
+    logger.info(u"Fetching Metacritic reviews for: %s" % artist['artist_name'])
+    metacritic.update(artistid, artist['artist_name'], artist['releasegroups'])
+
     if errors:
-        logger.info("[%s] Finished updating artist: %s but with errors, so not marking it as updated in the database" % (artist['artist_name'], artist['artist_name']))
+        logger.info(
+            "[%s] Finished updating artist: %s but with errors, so not marking it as updated in the database" % (
+            artist['artist_name'], artist['artist_name']))
     else:
         myDB.action('DELETE FROM newartists WHERE ArtistName = ?', [artist['artist_name']])
         logger.info(u"Updating complete for: %s" % artist['artist_name'])
@@ -504,37 +554,46 @@ def addArtisttoDB(artistid, extrasonly=False, forcefull=False):
         for album_search in album_searches:
             searcher.searchforalbum(albumid=album_search)
 
+
 def finalize_update(artistid, artistname, errors=False):
     # Moving this little bit to it's own function so we can update have tracks & latest album when deleting extras
 
     myDB = db.DBConnection()
 
-    latestalbum = myDB.action('SELECT AlbumTitle, ReleaseDate, AlbumID from albums WHERE ArtistID=? order by ReleaseDate DESC', [artistid]).fetchone()
-    totaltracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")', [artistid]))
-    #havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist['artist_name']]))
-    havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artistname]))
+    latestalbum = myDB.action(
+        'SELECT AlbumTitle, ReleaseDate, AlbumID from albums WHERE ArtistID=? order by ReleaseDate DESC',
+        [artistid]).fetchone()
+    totaltracks = len(myDB.select(
+        'SELECT TrackTitle from tracks WHERE ArtistID=? AND AlbumID IN (SELECT AlbumID FROM albums WHERE Status != "Ignored")',
+        [artistid]))
+    # havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [artistid])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ?', [artist['artist_name']]))
+    havetracks = len(
+        myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL',
+                    [artistid])) + len(
+        myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"',
+                    [artistname]))
 
-    controlValueDict = {"ArtistID":     artistid}
+    controlValueDict = {"ArtistID": artistid}
 
     if latestalbum:
-        newValueDict = {"Status":           "Active",
-                        "LatestAlbum":      latestalbum['AlbumTitle'],
-                        "ReleaseDate":      latestalbum['ReleaseDate'],
-                        "AlbumID":          latestalbum['AlbumID'],
-                        "TotalTracks":      totaltracks,
-                        "HaveTracks":       havetracks}
+        newValueDict = {"Status": "Active",
+                        "LatestAlbum": latestalbum['AlbumTitle'],
+                        "ReleaseDate": latestalbum['ReleaseDate'],
+                        "AlbumID": latestalbum['AlbumID'],
+                        "TotalTracks": totaltracks,
+                        "HaveTracks": havetracks}
     else:
-        newValueDict = {"Status":           "Active",
-                        "TotalTracks":      totaltracks,
-                        "HaveTracks":       havetracks}
+        newValueDict = {"Status": "Active",
+                        "TotalTracks": totaltracks,
+                        "HaveTracks": havetracks}
 
     if not errors:
         newValueDict['LastUpdated'] = helpers.now()
 
     myDB.upsert("artists", newValueDict, controlValueDict)
 
-def addReleaseById(rid, rgid=None):
 
+def addReleaseById(rid, rgid=None):
     myDB = db.DBConnection()
 
     # Create minimum info upfront if added from searchresults
@@ -543,24 +602,28 @@ def addReleaseById(rid, rgid=None):
         dbalbum = myDB.select("SELECT * from albums WHERE AlbumID=?", [rgid])
         if not dbalbum:
             status = 'Loading'
-            controlValueDict = {"AlbumID":  rgid}
-            newValueDict = {"AlbumTitle":   rgid,
-                            "ArtistName":   status,
-                            "Status":       status}
+            controlValueDict = {"AlbumID": rgid}
+            newValueDict = {"AlbumTitle": rgid,
+                            "ArtistName": status,
+                            "Status": status}
             myDB.upsert("albums", newValueDict, controlValueDict)
             time.sleep(1)
 
     rgid = None
     artistid = None
     release_dict = None
-    results = myDB.select("SELECT albums.ArtistID, releases.ReleaseGroupID from releases, albums WHERE releases.ReleaseID=? and releases.ReleaseGroupID=albums.AlbumID LIMIT 1", [rid])
+    results = myDB.select(
+        "SELECT albums.ArtistID, releases.ReleaseGroupID from releases, albums WHERE releases.ReleaseID=? and releases.ReleaseGroupID=albums.AlbumID LIMIT 1",
+        [rid])
     for result in results:
         rgid = result['ReleaseGroupID']
         artistid = result['ArtistID']
-        logger.debug("Found a cached releaseid : releasegroupid relationship: " + rid + " : " + rgid)
+        logger.debug(
+            "Found a cached releaseid : releasegroupid relationship: " + rid + " : " + rgid)
     if not rgid:
-        #didn't find it in the cache, get the information from MB
-        logger.debug("Didn't find releaseID " + rid + " in the cache. Looking up its ReleaseGroupID")
+        # didn't find it in the cache, get the information from MB
+        logger.debug(
+            "Didn't find releaseID " + rid + " in the cache. Looking up its ReleaseGroupID")
         try:
             release_dict = mb.getRelease(rid)
         except Exception as e:
@@ -577,10 +640,10 @@ def addReleaseById(rid, rgid=None):
         rgid = release_dict['rgid']
         artistid = release_dict['artist_id']
 
-    #we don't want to make more calls to MB here unless we have to, could be happening quite a lot
+    # we don't want to make more calls to MB here unless we have to, could be happening quite a lot
     rg_exists = myDB.select("SELECT * from albums WHERE AlbumID=?", [rgid])
 
-    #make sure the artist exists since I don't know what happens later if it doesn't
+    # make sure the artist exists since I don't know what happens later if it doesn't
     artist_exists = myDB.select("SELECT * from artists WHERE ArtistID=?", [artistid])
 
     if not artist_exists and release_dict:
@@ -589,97 +652,129 @@ def addReleaseById(rid, rgid=None):
         else:
             sortname = release_dict['artist_name']
 
-        logger.info(u"Now manually adding: " + release_dict['artist_name'] + " - with status Paused")
-        controlValueDict = {"ArtistID":     release_dict['artist_id']}
-        newValueDict = {"ArtistName":       release_dict['artist_name'],
-                        "ArtistSortName":   sortname,
-                        "DateAdded":        helpers.today(),
-                        "Status":           "Paused"}
+        logger.info(
+            u"Now manually adding: " + release_dict['artist_name'] + " - with status Paused")
+        controlValueDict = {"ArtistID": release_dict['artist_id']}
+        newValueDict = {"ArtistName": release_dict['artist_name'],
+                        "ArtistSortName": sortname,
+                        "DateAdded": helpers.today(),
+                        "Status": "Paused"}
 
-        if headphones.INCLUDE_EXTRAS:
+        if headphones.CONFIG.INCLUDE_EXTRAS:
             newValueDict['IncludeExtras'] = 1
-            newValueDict['Extras'] = headphones.EXTRAS
+            newValueDict['Extras'] = headphones.CONFIG.EXTRAS
+
+        if 'title' in release_dict:
+            newValueDict['LatestAlbum'] = release_dict['title']
+        elif 'rg_title' in release_dict:
+            newValueDict['LatestAlbum'] = release_dict['rg_title']
+
+        if 'date' in release_dict:
+            newValueDict['ReleaseDate'] = release_dict['date']
+
+        if rgid:
+            newValueDict['AlbumID'] = rgid
 
         myDB.upsert("artists", newValueDict, controlValueDict)
 
     elif not artist_exists and not release_dict:
-        logger.error("Artist does not exist in the database and did not get a valid response from MB. Skipping release.")
+        logger.error(
+            "Artist does not exist in the database and did not get a valid response from MB. Skipping release.")
         if status == 'Loading':
             myDB.action("DELETE FROM albums WHERE AlbumID=?", [rgid])
         return
 
-    if not rg_exists and release_dict or status == 'Loading' and release_dict:  #it should never be the case that we have an rg and not the artist
-                                                                                #but if it is this will fail
+    if not rg_exists and release_dict or status == 'Loading' and release_dict:  # it should never be the case that we have an rg and not the artist
+        # but if it is this will fail
         logger.info(u"Now adding-by-id album (" + release_dict['title'] + ") from id: " + rgid)
-        controlValueDict = {"AlbumID":  rgid}
+        controlValueDict = {"AlbumID": rgid}
         if status != 'Loading':
             status = 'Wanted'
 
-        newValueDict = {"ArtistID":         release_dict['artist_id'],
-                        "ReleaseID":        rgid,
-                        "ArtistName":       release_dict['artist_name'],
-                        "AlbumTitle":       release_dict['rg_title'],
-                        "AlbumASIN":        release_dict['asin'],
-                        "ReleaseDate":      release_dict['date'],
-                        "DateAdded":        helpers.today(),
-                        "Status":           status,
-                        "Type":             release_dict['rg_type'],
-                        "ReleaseID":        rid
+        newValueDict = {"ArtistID": release_dict['artist_id'],
+                        "ReleaseID": rgid,
+                        "ArtistName": release_dict['artist_name'],
+                        "AlbumTitle": release_dict['title'] if 'title' in release_dict else
+                        release_dict['rg_title'],
+                        "AlbumASIN": release_dict['asin'],
+                        "ReleaseDate": release_dict['date'],
+                        "DateAdded": helpers.today(),
+                        "Status": status,
+                        "Type": release_dict['rg_type'],
+                        "ReleaseID": rid
                         }
 
         myDB.upsert("albums", newValueDict, controlValueDict)
 
-        #keep a local cache of these so that external programs that are adding releasesByID don't hammer MB
+        # keep a local cache of these so that external programs that are adding releasesByID don't hammer MB
         myDB.action('INSERT INTO releases VALUES( ?, ?)', [rid, release_dict['rgid']])
 
         for track in release_dict['tracks']:
-            cleanname = helpers.cleanName(release_dict['artist_name'] + ' ' + release_dict['rg_title'] + ' ' + track['title'])
+            cleanname = helpers.cleanName(
+                release_dict['artist_name'] + ' ' + release_dict['rg_title'] + ' ' + track['title'])
 
-            controlValueDict = {"TrackID":  track['id'],
-                                "AlbumID":  rgid}
-            newValueDict = {"ArtistID":     release_dict['artist_id'],
-                        "ArtistName":       release_dict['artist_name'],
-                        "AlbumTitle":       release_dict['rg_title'],
-                        "AlbumASIN":        release_dict['asin'],
-                        "TrackTitle":       track['title'],
-                        "TrackDuration":    track['duration'],
-                        "TrackNumber":      track['number'],
-                        "CleanName":        cleanname
-                        }
+            controlValueDict = {"TrackID": track['id'],
+                                "AlbumID": rgid}
+            newValueDict = {"ArtistID": release_dict['artist_id'],
+                            "ArtistName": release_dict['artist_name'],
+                            "AlbumTitle": release_dict['rg_title'],
+                            "AlbumASIN": release_dict['asin'],
+                            "TrackTitle": track['title'],
+                            "TrackDuration": track['duration'],
+                            "TrackNumber": track['number'],
+                            "CleanName": cleanname
+                            }
 
-            match = myDB.action('SELECT Location, BitRate, Format from have WHERE CleanName=?', [cleanname]).fetchone()
+            match = myDB.action(
+                'SELECT Location, BitRate, Format, Matched from have WHERE CleanName=?',
+                [cleanname]).fetchone()
 
             if not match:
-                match = myDB.action('SELECT Location, BitRate, Format from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [release_dict['artist_name'], release_dict['rg_title'], track['title']]).fetchone()
+                match = myDB.action(
+                    'SELECT Location, BitRate, Format, Matched from have WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                    [release_dict['artist_name'], release_dict['rg_title'],
+                     track['title']]).fetchone()
 
-            #if not match:
-                #match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
+                # if not match:
+                # match = myDB.action('SELECT Location, BitRate, Format from have WHERE TrackID=?', [track['id']]).fetchone()
 
             if match:
                 newValueDict['Location'] = match['Location']
                 newValueDict['BitRate'] = match['BitRate']
                 newValueDict['Format'] = match['Format']
-                #myDB.action('DELETE from have WHERE Location=?', [match['Location']])
+                # myDB.action('DELETE from have WHERE Location=?', [match['Location']])
+
+                # If the album has been scanned before adding the release it will be unmatched, update to matched
+                if match['Matched'] == 'Failed':
+                    myDB.action('UPDATE have SET Matched=? WHERE Location=?',
+                                (release_dict['rgid'], match['Location']))
 
             myDB.upsert("tracks", newValueDict, controlValueDict)
 
         # Reset status
         if status == 'Loading':
-            controlValueDict = {"AlbumID":  rgid}
-            newValueDict = {"Status":   "Wanted"}
+            controlValueDict = {"AlbumID": rgid}
+            if headphones.CONFIG.AUTOWANT_MANUALLY_ADDED:
+                newValueDict = {"Status": "Wanted"}
+            else:
+                newValueDict = {"Status": "Skipped"}
             myDB.upsert("albums", newValueDict, controlValueDict)
 
         # Start a search for the album
-        import searcher
-        searcher.searchforalbum(rgid, False)
+        if headphones.CONFIG.AUTOWANT_MANUALLY_ADDED:
+            import searcher
+            searcher.searchforalbum(rgid, False)
+
     elif not rg_exists and not release_dict:
-        logger.error("ReleaseGroup does not exist in the database and did not get a valid response from MB. Skipping release.")
+        logger.error(
+            "ReleaseGroup does not exist in the database and did not get a valid response from MB. Skipping release.")
         if status == 'Loading':
             myDB.action("DELETE FROM albums WHERE AlbumID=?", [rgid])
         return
     else:
         logger.info('Release ' + str(rid) + " already exists in the database!")
 
+
 def updateFormat():
     myDB = db.DBConnection()
     tracks = myDB.select('SELECT * from tracks WHERE Location IS NOT NULL and Format IS NULL')
@@ -688,10 +783,10 @@ def updateFormat():
         for track in tracks:
             try:
                 f = MediaFile(track['Location'])
-            except Exception, e:
+            except Exception as e:
                 logger.info("Exception from MediaFile for: " + track['Location'] + " : " + str(e))
                 continue
-            controlValueDict = {"TrackID":  track['TrackID']}
+            controlValueDict = {"TrackID": track['TrackID']}
             newValueDict = {"Format": f.format}
             myDB.upsert("tracks", newValueDict, controlValueDict)
         logger.info('Finished finding media format for %s files' % len(tracks))
@@ -701,14 +796,15 @@ def updateFormat():
         for track in havetracks:
             try:
                 f = MediaFile(track['Location'])
-            except Exception, e:
+            except Exception as e:
                 logger.info("Exception from MediaFile for: " + track['Location'] + " : " + str(e))
                 continue
-            controlValueDict = {"TrackID":  track['TrackID']}
+            controlValueDict = {"TrackID": track['TrackID']}
             newValueDict = {"Format": f.format}
             myDB.upsert("have", newValueDict, controlValueDict)
         logger.info('Finished finding media format for %s files' % len(havetracks))
 
+
 def getHybridRelease(fullreleaselist):
     """
     Returns a dictionary of best group of tracks from the list of releases and
@@ -721,18 +817,18 @@ def getHybridRelease(fullreleaselist):
     sortable_release_list = []
 
     formats = {
-        '2xVinyl':          '2',
-        'Vinyl':            '2',
-        'CD':               '0',
-        'Cassette':         '3',
-        '2xCD':             '1',
-        'Digital Media':    '0'
+        '2xVinyl': '2',
+        'Vinyl': '2',
+        'CD': '0',
+        'Cassette': '3',
+        '2xCD': '1',
+        'Digital Media': '0'
     }
 
     countries = {
-        'US':    '0',
-        'GB':    '1',
-        'JP':    '2',
+        'US': '0',
+        'GB': '1',
+        'JP': '2',
     }
 
     for release in fullreleaselist:
@@ -749,14 +845,14 @@ def getHybridRelease(fullreleaselist):
 
         # Create record
         release_dict = {
-            'hasasin':        bool(release['AlbumASIN']),
-            'asin':           release['AlbumASIN'],
-            'trackscount':    len(release['Tracks']),
-            'releaseid':      release['ReleaseID'],
-            'releasedate':    release['ReleaseDate'],
-            'format':         format,
-            'country':        country,
-            'tracks':         release['Tracks']
+            'hasasin': bool(release['AlbumASIN']),
+            'asin': release['AlbumASIN'],
+            'trackscount': len(release['Tracks']),
+            'releaseid': release['ReleaseID'],
+            'releasedate': release['ReleaseDate'],
+            'format': format,
+            'country': country,
+            'tracks': release['Tracks']
         }
 
         sortable_release_list.append(release_dict)
@@ -767,8 +863,8 @@ def getHybridRelease(fullreleaselist):
         # Change this value to change the sorting behaviour of none, returning
         # 'None' will put it at the top which was normal behaviour for pre-ngs
         # versions
-        if releaseDate == None:
-            return 'None';
+        if releaseDate is None:
+            return 'None'
 
         if releaseDate.count('-') == 2:
             return releaseDate
@@ -777,17 +873,19 @@ def getHybridRelease(fullreleaselist):
         else:
             return releaseDate + '13-32'
 
-    sortable_release_list.sort(key=lambda x:getSortableReleaseDate(x['releasedate']))
+    sortable_release_list.sort(key=lambda x: getSortableReleaseDate(x['releasedate']))
 
-    average_tracks = sum(x['trackscount'] for x in sortable_release_list) / float(len(sortable_release_list))
+    average_tracks = sum(x['trackscount'] for x in sortable_release_list) / float(
+        len(sortable_release_list))
     for item in sortable_release_list:
         item['trackscount_delta'] = abs(average_tracks - item['trackscount'])
 
-    a = helpers.multikeysort(sortable_release_list, ['-hasasin', 'country', 'format', 'trackscount_delta'])
+    a = helpers.multikeysort(sortable_release_list,
+                             ['-hasasin', 'country', 'format', 'trackscount_delta'])
 
-    release_dict = {'ReleaseDate'    : sortable_release_list[0]['releasedate'],
-                    'Tracks'         : a[0]['tracks'],
-                    'AlbumASIN'      : a[0]['asin']
+    release_dict = {'ReleaseDate': sortable_release_list[0]['releasedate'],
+                    'Tracks': a[0]['tracks'],
+                    'AlbumASIN': a[0]['asin']
                     }
 
     return release_dict

headphones/lastfm.py

@@ -14,17 +14,21 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import random
-import time
-import headphones
-
-from headphones import db, logger, request
-
 from collections import defaultdict
 
-TIMEOUT = 60 # seconds
+import headphones
+import headphones.lock
+from headphones import db, logger, request
+
+TIMEOUT = 60.0  # seconds
+REQUEST_LIMIT = 1.0 / 5  # seconds
 ENTRY_POINT = "http://ws.audioscrobbler.com/2.0/"
 API_KEY = "395e6ec6bb557382fc41fde867bce66f"
 
+# Required for API request limit
+lastfm_lock = headphones.lock.TimedLock(REQUEST_LIMIT)
+
+
 def request_lastfm(method, **kwargs):
     """
     Call a Last.FM API method. Automatically sets the method and API key. Method
@@ -43,7 +47,7 @@ def request_lastfm(method, **kwargs):
     logger.debug("Calling Last.FM method: %s", method)
     logger.debug("Last.FM call parameters: %s", kwargs)
 
-    data = request.request_json(ENTRY_POINT, timeout=TIMEOUT, params=kwargs)
+    data = request.request_json(ENTRY_POINT, timeout=TIMEOUT, params=kwargs, lock=lastfm_lock)
 
     # Parse response and check for errors.
     if not data:
@@ -56,6 +60,7 @@ def request_lastfm(method, **kwargs):
 
     return data
 
+
 def getSimilar():
     myDB = db.DBConnection()
     results = myDB.select("SELECT ArtistID from artists ORDER BY HaveTracks DESC")
@@ -65,7 +70,6 @@ def getSimilar():
 
     for result in results[:12]:
         data = request_lastfm("artist.getsimilar", mbid=result["ArtistId"])
-        time.sleep(10)
 
         if data and "similarartists" in data:
             artists = data["similarartists"]["artist"]
@@ -74,7 +78,7 @@ def getSimilar():
                 try:
                     artist_mbid = artist["mbid"]
                     artist_name = artist["name"]
-                except TypeError:
+                except KeyError:
                     continue
 
                 if not any(artist_mbid in x for x in results):
@@ -101,16 +105,17 @@ def getSimilar():
 
     logger.debug("Inserted %d artists into Last.FM tag cloud", len(top_list))
 
+
 def getArtists():
     myDB = db.DBConnection()
     results = myDB.select("SELECT ArtistID from artists")
 
-    if not headphones.LASTFM_USERNAME:
+    if not headphones.CONFIG.LASTFM_USERNAME:
         logger.warn("Last.FM username not set, not importing artists.")
         return
 
-    logger.info("Fetching artists from Last.FM for username: %s", headphones.LASTFM_USERNAME)
-    data = request_lastfm("library.getartists", limit=10000, user=headphones.LASTFM_USERNAME)
+    logger.info("Fetching artists from Last.FM for username: %s", headphones.CONFIG.LASTFM_USERNAME)
+    data = request_lastfm("library.getartists", limit=1000, user=headphones.CONFIG.LASTFM_USERNAME)
 
     if data and "artists" in data:
         artistlist = []
@@ -130,6 +135,7 @@ def getArtists():
 
         logger.info("Imported %d new artists from Last.FM", len(artistlist))
 
+
 def getTagTopArtists(tag, limit=50):
     myDB = db.DBConnection()
     results = myDB.select("SELECT ArtistID from artists")
@@ -153,4 +159,4 @@ def getTagTopArtists(tag, limit=50):
         for artistid in artistlist:
             importer.addArtisttoDB(artistid)
 
-        logger.debug("Added %d new artists from Last.FM", len(artistlist))
+        logger.debug("Added %d new artists from Last.FM", len(artistlist))

headphones/librarysync.py

@@ -14,33 +14,34 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import glob
+import math
+
 import headphones
-
 from beets.mediafile import MediaFile, FileTypeError, UnreadableFileError
-
 from headphones import db, logger, helpers, importer, lastfm
 
-# You can scan a single directory and append it to the current library by specifying append=True, ArtistID & ArtistName
-def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=False):
 
-
-    if cron and not headphones.LIBRARYSCAN:
+# You can scan a single directory and append it to the current library by
+# specifying append=True, ArtistID and ArtistName.
+def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None,
+                cron=False, artistScan=False):
+    if cron and not headphones.CONFIG.LIBRARYSCAN:
         return
 
     if not dir:
-        if not headphones.MUSIC_DIR:
+        if not headphones.CONFIG.MUSIC_DIR:
             return
         else:
-            dir = headphones.MUSIC_DIR
+            dir = headphones.CONFIG.MUSIC_DIR
 
     # If we're appending a dir, it's coming from the post processor which is
     # already bytestring
-    if not append:
+    if not append or artistScan:
         dir = dir.encode(headphones.SYS_ENCODING)
 
     if not os.path.isdir(dir):
-        logger.warn('Cannot find directory: %s. Not scanning' % dir.decode(headphones.SYS_ENCODING, 'replace'))
+        logger.warn('Cannot find directory: %s. Not scanning' % dir.decode(headphones.SYS_ENCODING,
+                                                                           'replace'))
         return
 
     myDB = db.DBConnection()
@@ -50,13 +51,16 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
 
     if not append:
         # Clean up bad filepaths
-        tracks = myDB.select('SELECT Location from alltracks WHERE Location IS NOT NULL UNION SELECT Location from tracks WHERE Location IS NOT NULL')
+        tracks = myDB.select(
+            'SELECT Location from alltracks WHERE Location IS NOT NULL UNION SELECT Location from tracks WHERE Location IS NOT NULL')
 
         for track in tracks:
-            encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING)
+            encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING, 'replace')
             if not os.path.isfile(encoded_track_string):
-                myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, track['Location']])
-                myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, track['Location']])
+                myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, track['Location']])
+                myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, track['Location']])
 
         del_have_tracks = myDB.select('SELECT Location, Matched, ArtistName from have')
 
@@ -64,37 +68,41 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
             encoded_track_string = track['Location'].encode(headphones.SYS_ENCODING, 'replace')
             if not os.path.isfile(encoded_track_string):
                 if track['ArtistName']:
-                    #Make sure deleted files get accounted for when updating artist track counts
+                    # Make sure deleted files get accounted for when updating artist track counts
                     new_artists.append(track['ArtistName'])
                 myDB.action('DELETE FROM have WHERE Location=?', [track['Location']])
-                logger.info('File %s removed from Headphones, as it is no longer on disk' % encoded_track_string.decode(headphones.SYS_ENCODING, 'replace'))
-           ###############myDB.action('DELETE from have')
+                logger.info(
+                    'File %s removed from Headphones, as it is no longer on disk' % encoded_track_string.decode(
+                        headphones.SYS_ENCODING, 'replace'))
 
     bitrates = []
-
     song_list = []
+    latest_subdirectory = []
+
     new_song_count = 0
     file_count = 0
 
-    latest_subdirectory = []
-
-    for r,d,f in os.walk(dir):
-        #need to abuse slicing to get a copy of the list, doing it directly will skip the element after a deleted one
-        #using a list comprehension will not work correctly for nested subdirectories (os.walk keeps its original list)
-        for directory in d[:]:
-            if directory.startswith("."):
-                d.remove(directory)
+    for r, d, f in helpers.walk_directory(dir):
+        # Filter paths based on config. Note that these methods work directly
+        # on the inputs
+        helpers.path_filter_patterns(d, headphones.CONFIG.IGNORED_FOLDERS, r)
+        helpers.path_filter_patterns(f, headphones.CONFIG.IGNORED_FILES, r)
 
         for files in f:
             # MEDIA_FORMATS = music file extensions, e.g. mp3, flac, etc
             if any(files.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
-
-                subdirectory = r.replace(dir,'')
+                subdirectory = r.replace(dir, '')
                 latest_subdirectory.append(subdirectory)
-                if file_count == 0 and r.replace(dir,'') !='':
-                    logger.info("[%s] Now scanning subdirectory %s" % (dir.decode(headphones.SYS_ENCODING, 'replace'), subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
-                elif latest_subdirectory[file_count] != latest_subdirectory[file_count-1] and file_count !=0:
-                    logger.info("[%s] Now scanning subdirectory %s" % (dir.decode(headphones.SYS_ENCODING, 'replace'), subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
+
+                if file_count == 0 and r.replace(dir, '') != '':
+                    logger.info("[%s] Now scanning subdirectory %s" % (
+                        dir.decode(headphones.SYS_ENCODING, 'replace'),
+                        subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
+                elif latest_subdirectory[file_count] != latest_subdirectory[
+                            file_count - 1] and file_count != 0:
+                    logger.info("[%s] Now scanning subdirectory %s" % (
+                        dir.decode(headphones.SYS_ENCODING, 'replace'),
+                        subdirectory.decode(headphones.SYS_ENCODING, 'replace')))
 
                 song = os.path.join(r, files)
 
@@ -105,7 +113,13 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
                 try:
                     f = MediaFile(song)
                 except (FileTypeError, UnreadableFileError):
-                    logger.error("Cannot read file media file '%s'. It may be corrupted or not a media file.", unicode_song_path)
+                    logger.warning(
+                        "Cannot read media file '%s', skipping. It may be corrupted or not a media file.",
+                        unicode_song_path)
+                    continue
+                except IOError:
+                    logger.warning("Cannnot read media file '%s', skipping. Does the file exists?",
+                                   unicode_song_path)
                     continue
 
                 # Grab the bitrates for the auto detect bit rate option
@@ -124,63 +138,73 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
                 # TODO: skip adding songs without the minimum requisite information (just a matter of putting together the right if statements)
 
                 if f_artist and f.album and f.title:
-                    CleanName = helpers.cleanName(f_artist +' '+ f.album +' '+ f.title)
+                    CleanName = helpers.cleanName(f_artist + ' ' + f.album + ' ' + f.title)
                 else:
                     CleanName = None
 
-                controlValueDict = {'Location'   : unicode_song_path}
+                controlValueDict = {'Location': unicode_song_path}
 
-                newValueDict = { 'TrackID' : f.mb_trackid,
-                              #'ReleaseID' : f.mb_albumid,
-                              'ArtistName' : f_artist,
-                              'AlbumTitle' : f.album,
-                              'TrackNumber': f.track,
-                              'TrackLength': f.length,
-                              'Genre'      : f.genre,
-                              'Date'       : f.date,
-                              'TrackTitle' : f.title,
-                              'BitRate'    : f.bitrate,
-                              'Format'     : f.format,
-                              'CleanName'  : CleanName
-                              }
+                newValueDict = {'TrackID': f.mb_trackid,
+                                # 'ReleaseID' : f.mb_albumid,
+                                'ArtistName': f_artist,
+                                'AlbumTitle': f.album,
+                                'TrackNumber': f.track,
+                                'TrackLength': f.length,
+                                'Genre': f.genre,
+                                'Date': f.date,
+                                'TrackTitle': f.title,
+                                'BitRate': f.bitrate,
+                                'Format': f.format,
+                                'CleanName': CleanName
+                                }
 
-                #song_list.append(song_dict)
-                check_exist_song = myDB.action("SELECT * FROM have WHERE Location=?", [unicode_song_path]).fetchone()
-                #Only attempt to match songs that are new, haven't yet been matched, or metadata has changed.
+                # song_list.append(song_dict)
+                check_exist_song = myDB.action("SELECT * FROM have WHERE Location=?",
+                                               [unicode_song_path]).fetchone()
+                # Only attempt to match songs that are new, haven't yet been matched, or metadata has changed.
                 if not check_exist_song:
-                    #This is a new track
+                    # This is a new track
                     if f_artist:
                         new_artists.append(f_artist)
                     myDB.upsert("have", newValueDict, controlValueDict)
-                    new_song_count+=1
+                    new_song_count += 1
                 else:
-                    if check_exist_song['ArtistName'] != f_artist or check_exist_song['AlbumTitle'] != f.album or check_exist_song['TrackTitle'] != f.title:
-                        #Important track metadata has been modified, need to run matcher again
+                    if check_exist_song['ArtistName'] != f_artist or check_exist_song[
+                        'AlbumTitle'] != f.album or check_exist_song['TrackTitle'] != f.title:
+                        # Important track metadata has been modified, need to run matcher again
                         if f_artist and f_artist != check_exist_song['ArtistName']:
                             new_artists.append(f_artist)
-                        elif f_artist and f_artist == check_exist_song['ArtistName'] and check_exist_song['Matched'] != "Ignored":
+                        elif f_artist and f_artist == check_exist_song['ArtistName'] and \
+                                        check_exist_song['Matched'] != "Ignored":
                             new_artists.append(f_artist)
                         else:
                             continue
 
                         newValueDict['Matched'] = None
                         myDB.upsert("have", newValueDict, controlValueDict)
-                        myDB.action('UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, unicode_song_path])
-                        myDB.action('UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?', [None, None, None, unicode_song_path])
-                        new_song_count+=1
+                        myDB.action(
+                            'UPDATE tracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, unicode_song_path])
+                        myDB.action(
+                            'UPDATE alltracks SET Location=?, BitRate=?, Format=? WHERE Location=?',
+                            [None, None, None, unicode_song_path])
+                        new_song_count += 1
                     else:
-                        #This track information hasn't changed
+                        # This track information hasn't changed
                         if f_artist and check_exist_song['Matched'] != "Ignored":
                             new_artists.append(f_artist)
 
-                file_count+=1
-
+                file_count += 1
 
     # Now we start track matching
     logger.info("%s new/modified songs found and added to the database" % new_song_count)
-    song_list = myDB.action("SELECT * FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace')+"%"])
-    total_number_of_songs = myDB.action("SELECT COUNT(*) FROM have WHERE Matched IS NULL AND LOCATION LIKE ?", [dir.decode(headphones.SYS_ENCODING, 'replace')+"%"]).fetchone()[0]
-    logger.info("Found " + str(total_number_of_songs) + " new/modified tracks in: '" + dir.decode(headphones.SYS_ENCODING, 'replace') + "'. Matching tracks to the appropriate releases....")
+    song_list = myDB.action("SELECT * FROM have WHERE Matched IS NULL AND LOCATION LIKE ?",
+                            [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"])
+    total_number_of_songs = \
+        myDB.action("SELECT COUNT(*) FROM have WHERE Matched IS NULL AND LOCATION LIKE ?",
+                    [dir.decode(headphones.SYS_ENCODING, 'replace') + "%"]).fetchone()[0]
+    logger.info("Found " + str(total_number_of_songs) + " new/modified tracks in: '" + dir.decode(
+        headphones.SYS_ENCODING, 'replace') + "'. Matching tracks to the appropriate releases....")
 
     # Sort the song_list by most vague (e.g. no trackid or releaseid) to most specific (both trackid & releaseid)
     # When we insert into the database, the tracks with the most specific information will overwrite the more general matches
@@ -191,183 +215,220 @@ def libraryScan(dir=None, append=False, ArtistID=None, ArtistName=None, cron=Fal
     # We'll use this to give a % completion, just because the track matching might take a while
     song_count = 0
     latest_artist = []
+    last_completion_percentage = 0
 
     for song in song_list:
 
         latest_artist.append(song['ArtistName'])
         if song_count == 0:
             logger.info("Now matching songs by %s" % song['ArtistName'])
-        elif latest_artist[song_count] != latest_artist[song_count-1] and song_count !=0:
+        elif latest_artist[song_count] != latest_artist[song_count - 1] and song_count != 0:
             logger.info("Now matching songs by %s" % song['ArtistName'])
 
-        #print song['ArtistName']+' - '+song['AlbumTitle']+' - '+song['TrackTitle']
         song_count += 1
-        completion_percentage = float(song_count)/total_number_of_songs * 100
+        completion_percentage = math.floor(float(song_count) / total_number_of_songs * 1000) / 10
 
-        if completion_percentage%10 == 0:
+        if completion_percentage >= (last_completion_percentage + 10):
             logger.info("Track matching is " + str(completion_percentage) + "% complete")
+            last_completion_percentage = completion_percentage
 
-        #THE "MORE-SPECIFIC" CLAUSES HERE HAVE ALL BEEN REMOVED.  WHEN RUNNING A LIBRARY SCAN, THE ONLY CLAUSES THAT
-        #EVER GOT HIT WERE [ARTIST/ALBUM/TRACK] OR CLEANNAME.  ARTISTID & RELEASEID ARE NEVER PASSED TO THIS FUNCTION,
-        #ARE NEVER FOUND, AND THE OTHER CLAUSES WERE NEVER HIT.  FURTHERMORE, OTHER MATCHING FUNCTIONS IN THIS PROGRAM
-        #(IMPORTER.PY, MB.PY) SIMPLY DO A [ARTIST/ALBUM/TRACK] OR CLEANNAME MATCH, SO IT'S ALL CONSISTENT.
+        # THE "MORE-SPECIFIC" CLAUSES HERE HAVE ALL BEEN REMOVED.  WHEN RUNNING A LIBRARY SCAN, THE ONLY CLAUSES THAT
+        # EVER GOT HIT WERE [ARTIST/ALBUM/TRACK] OR CLEANNAME.  ARTISTID & RELEASEID ARE NEVER PASSED TO THIS FUNCTION,
+        # ARE NEVER FOUND, AND THE OTHER CLAUSES WERE NEVER HIT.  FURTHERMORE, OTHER MATCHING FUNCTIONS IN THIS PROGRAM
+        # (IMPORTER.PY, MB.PY) SIMPLY DO A [ARTIST/ALBUM/TRACK] OR CLEANNAME MATCH, SO IT'S ALL CONSISTENT.
 
         if song['ArtistName'] and song['AlbumTitle'] and song['TrackTitle']:
 
-            track = myDB.action('SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from tracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
+            track = myDB.action(
+                'SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from tracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
             have_updated = False
             if track:
-                controlValueDict = { 'ArtistName' : track['ArtistName'],
-                                     'AlbumTitle' : track['AlbumTitle'],
-                                     'TrackTitle' : track['TrackTitle'] }
-                newValueDict = { 'Location' : song['Location'],
-                                 'BitRate'  : song['BitRate'],
-                                 'Format'   : song['Format'] }
+                controlValueDict = {'ArtistName': track['ArtistName'],
+                                    'AlbumTitle': track['AlbumTitle'],
+                                    'TrackTitle': track['TrackTitle']}
+                newValueDict = {'Location': song['Location'],
+                                'BitRate': song['BitRate'],
+                                'Format': song['Format']}
                 myDB.upsert("tracks", newValueDict, controlValueDict)
 
-                controlValueDict2 = { 'Location' : song['Location']}
-                newValueDict2 = { 'Matched' : track['AlbumID']}
+                controlValueDict2 = {'Location': song['Location']}
+                newValueDict2 = {'Matched': track['AlbumID']}
                 myDB.upsert("have", newValueDict2, controlValueDict2)
                 have_updated = True
             else:
-                track = myDB.action('SELECT CleanName, AlbumID from tracks WHERE CleanName LIKE ?', [song['CleanName']]).fetchone()
+                track = myDB.action('SELECT CleanName, AlbumID from tracks WHERE CleanName LIKE ?',
+                                    [song['CleanName']]).fetchone()
                 if track:
-                    controlValueDict = { 'CleanName' : track['CleanName']}
-                    newValueDict = { 'Location' : song['Location'],
-                                     'BitRate'  : song['BitRate'],
-                                     'Format'   : song['Format'] }
+                    controlValueDict = {'CleanName': track['CleanName']}
+                    newValueDict = {'Location': song['Location'],
+                                    'BitRate': song['BitRate'],
+                                    'Format': song['Format']}
                     myDB.upsert("tracks", newValueDict, controlValueDict)
 
-                    controlValueDict2 = { 'Location' : song['Location']}
-                    newValueDict2 = { 'Matched' : track['AlbumID']}
+                    controlValueDict2 = {'Location': song['Location']}
+                    newValueDict2 = {'Matched': track['AlbumID']}
                     myDB.upsert("have", newValueDict2, controlValueDict2)
                     have_updated = True
                 else:
-                    controlValueDict2 = { 'Location' : song['Location']}
-                    newValueDict2 = { 'Matched' : "Failed"}
+                    controlValueDict2 = {'Location': song['Location']}
+                    newValueDict2 = {'Matched': "Failed"}
                     myDB.upsert("have", newValueDict2, controlValueDict2)
                     have_updated = True
 
-            alltrack = myDB.action('SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from alltracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?', [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
+            alltrack = myDB.action(
+                'SELECT ArtistName, AlbumTitle, TrackTitle, AlbumID from alltracks WHERE ArtistName LIKE ? AND AlbumTitle LIKE ? AND TrackTitle LIKE ?',
+                [song['ArtistName'], song['AlbumTitle'], song['TrackTitle']]).fetchone()
             if alltrack:
-                controlValueDict = { 'ArtistName' : alltrack['ArtistName'],
-                                     'AlbumTitle' : alltrack['AlbumTitle'],
-                                     'TrackTitle' : alltrack['TrackTitle'] }
-                newValueDict = { 'Location' : song['Location'],
-                                 'BitRate'  : song['BitRate'],
-                                 'Format'   : song['Format'] }
+                controlValueDict = {'ArtistName': alltrack['ArtistName'],
+                                    'AlbumTitle': alltrack['AlbumTitle'],
+                                    'TrackTitle': alltrack['TrackTitle']}
+                newValueDict = {'Location': song['Location'],
+                                'BitRate': song['BitRate'],
+                                'Format': song['Format']}
                 myDB.upsert("alltracks", newValueDict, controlValueDict)
 
-                controlValueDict2 = { 'Location' : song['Location']}
-                newValueDict2 = { 'Matched' : alltrack['AlbumID']}
+                controlValueDict2 = {'Location': song['Location']}
+                newValueDict2 = {'Matched': alltrack['AlbumID']}
                 myDB.upsert("have", newValueDict2, controlValueDict2)
             else:
-                alltrack = myDB.action('SELECT CleanName, AlbumID from alltracks WHERE CleanName LIKE ?', [song['CleanName']]).fetchone()
+                alltrack = myDB.action(
+                    'SELECT CleanName, AlbumID from alltracks WHERE CleanName LIKE ?',
+                    [song['CleanName']]).fetchone()
                 if alltrack:
-                    controlValueDict = { 'CleanName' : alltrack['CleanName']}
-                    newValueDict = { 'Location' : song['Location'],
-                                     'BitRate'  : song['BitRate'],
-                                     'Format'   : song['Format'] }
+                    controlValueDict = {'CleanName': alltrack['CleanName']}
+                    newValueDict = {'Location': song['Location'],
+                                    'BitRate': song['BitRate'],
+                                    'Format': song['Format']}
                     myDB.upsert("alltracks", newValueDict, controlValueDict)
 
-                    controlValueDict2 = { 'Location' : song['Location']}
-                    newValueDict2 = { 'Matched' : alltrack['AlbumID']}
+                    controlValueDict2 = {'Location': song['Location']}
+                    newValueDict2 = {'Matched': alltrack['AlbumID']}
                     myDB.upsert("have", newValueDict2, controlValueDict2)
                 else:
                     # alltracks may not exist if adding album manually, have should only be set to failed if not already updated in tracks
                     if not have_updated:
-                        controlValueDict2 = { 'Location' : song['Location']}
-                        newValueDict2 = { 'Matched' : "Failed"}
+                        controlValueDict2 = {'Location': song['Location']}
+                        newValueDict2 = {'Matched': "Failed"}
                         myDB.upsert("have", newValueDict2, controlValueDict2)
 
         else:
-            controlValueDict2 = { 'Location' : song['Location']}
-            newValueDict2 = { 'Matched' : "Failed"}
+            controlValueDict2 = {'Location': song['Location']}
+            newValueDict2 = {'Matched': "Failed"}
             myDB.upsert("have", newValueDict2, controlValueDict2)
 
-        #######myDB.action('INSERT INTO have (ArtistName, AlbumTitle, TrackNumber, TrackTitle, TrackLength, BitRate, Genre, Date, TrackID, Location, CleanName, Format) VALUES( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', [song['ArtistName'], song['AlbumTitle'], song['TrackNumber'], song['TrackTitle'], song['TrackLength'], song['BitRate'], song['Genre'], song['Date'], song['TrackID'], song['Location'], CleanName, song['Format']])
+            #######myDB.action('INSERT INTO have (ArtistName, AlbumTitle, TrackNumber, TrackTitle, TrackLength, BitRate, Genre, Date, TrackID, Location, CleanName, Format) VALUES( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', [song['ArtistName'], song['AlbumTitle'], song['TrackNumber'], song['TrackTitle'], song['TrackLength'], song['BitRate'], song['Genre'], song['Date'], song['TrackID'], song['Location'], CleanName, song['Format']])
 
-    logger.info('Completed matching tracks from directory: %s' % dir.decode(headphones.SYS_ENCODING, 'replace'))
+    logger.info('Completed matching tracks from directory: %s' % dir.decode(headphones.SYS_ENCODING,
+                                                                            'replace'))
 
-
-    if not append:
+    if not append or artistScan:
         logger.info('Updating scanned artist track counts')
 
         # Clean up the new artist list
         unique_artists = {}.fromkeys(new_artists).keys()
         current_artists = myDB.select('SELECT ArtistName, ArtistID from artists')
 
-        #There was a bug where artists with special characters (-,') would show up in new artists.
-        artist_list = [f for f in unique_artists if helpers.cleanName(f).lower() not in [helpers.cleanName(x[0]).lower() for x in current_artists]]
-        artists_checked = [f for f in unique_artists if helpers.cleanName(f).lower() in [helpers.cleanName(x[0]).lower() for x in current_artists]]
+        # There was a bug where artists with special characters (-,') would show up in new artists.
+        artist_list = [
+            x for x in unique_artists
+            if helpers.cleanName(x).lower() not in [
+                helpers.cleanName(y[0]).lower()
+                for y in current_artists
+                ]
+            ]
+        artists_checked = [
+            x for x in unique_artists
+            if helpers.cleanName(x).lower() in [
+                helpers.cleanName(y[0]).lower()
+                for y in current_artists
+                ]
+            ]
 
         # Update track counts
-
         for artist in artists_checked:
             # Have tracks are selected from tracks table and not all tracks because of duplicates
             # We update the track count upon an album switch to compliment this
-            havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL', [artist])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [artist]))
-            #Note, some people complain about having "artist have tracks" > # of tracks total in artist official releases
+            havetracks = (
+                len(myDB.select(
+                    'SELECT TrackTitle from tracks WHERE ArtistName like ? AND Location IS NOT NULL',
+                    [artist])) + len(myDB.select(
+                    'SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"',
+                    [artist]))
+            )
+            # Note: some people complain about having "artist have tracks" > # of tracks total in artist official releases
             # (can fix by getting rid of second len statement)
             myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistName=?', [havetracks, artist])
 
         logger.info('Found %i new artists' % len(artist_list))
 
-        if len(artist_list):
-            if headphones.ADD_ARTISTS:
+        if artist_list:
+            if headphones.CONFIG.AUTO_ADD_ARTISTS:
                 logger.info('Importing %i new artists' % len(artist_list))
                 importer.artistlist_to_mbids(artist_list)
             else:
                 logger.info('To add these artists, go to Manage->Manage New Artists')
-                #myDB.action('DELETE from newartists')
+                # myDB.action('DELETE from newartists')
                 for artist in artist_list:
                     myDB.action('INSERT OR IGNORE INTO newartists VALUES (?)', [artist])
 
-        if headphones.DETECT_BITRATE:
-            headphones.PREFERRED_BITRATE = sum(bitrates)/len(bitrates)/1000
+        if headphones.CONFIG.DETECT_BITRATE and bitrates:
+            headphones.CONFIG.PREFERRED_BITRATE = sum(bitrates) / len(bitrates) / 1000
 
     else:
         # If we're appending a new album to the database, update the artists total track counts
         logger.info('Updating artist track counts')
 
-        havetracks = len(myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL', [ArtistID])) + len(myDB.select('SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"', [ArtistName]))
+        havetracks = len(
+            myDB.select('SELECT TrackTitle from tracks WHERE ArtistID=? AND Location IS NOT NULL',
+                        [ArtistID])) + len(myDB.select(
+            'SELECT TrackTitle from have WHERE ArtistName like ? AND Matched = "Failed"',
+            [ArtistName]))
         myDB.action('UPDATE artists SET HaveTracks=? WHERE ArtistID=?', [havetracks, ArtistID])
 
     if not append:
         update_album_status()
+
+    if not append and not artistScan:
         lastfm.getSimilar()
+
     logger.info('Library scan complete')
 
-#ADDED THIS SECTION TO MARK ALBUMS AS DOWNLOADED IF ARTISTS ARE ADDED EN MASSE BEFORE LIBRARY IS SCANNED
+
+# ADDED THIS SECTION TO MARK ALBUMS AS DOWNLOADED IF ARTISTS ARE ADDED EN MASSE BEFORE LIBRARY IS SCANNED
+
+
 def update_album_status(AlbumID=None):
     myDB = db.DBConnection()
     logger.info('Counting matched tracks to mark albums as skipped/downloaded')
     if AlbumID:
-        album_status_updater = myDB.action('SELECT AlbumID, AlbumTitle, Status from albums WHERE AlbumID=?', [AlbumID])
+        album_status_updater = myDB.action(
+            'SELECT AlbumID, AlbumTitle, Status from albums WHERE AlbumID=?', [AlbumID])
     else:
         album_status_updater = myDB.action('SELECT AlbumID, AlbumTitle, Status from albums')
     for album in album_status_updater:
-        track_counter = myDB.action('SELECT Location from tracks where AlbumID=?', [album['AlbumID']])
+        track_counter = myDB.action('SELECT Location from tracks where AlbumID=?',
+                                    [album['AlbumID']])
         total_tracks = 0
         have_tracks = 0
         for track in track_counter:
-            total_tracks+=1
+            total_tracks += 1
             if track['Location']:
-                have_tracks+=1
+                have_tracks += 1
         if total_tracks != 0:
             album_completion = float(float(have_tracks) / float(total_tracks)) * 100
         else:
             album_completion = 0
             logger.info('Album %s does not have any tracks in database' % album['AlbumTitle'])
 
-        if album_completion >= headphones.ALBUM_COMPLETION_PCT and album['Status'] == 'Skipped':
+        if album_completion >= headphones.CONFIG.ALBUM_COMPLETION_PCT:
             new_album_status = "Downloaded"
 
         # I don't think we want to change Downloaded->Skipped.....
         # I think we can only automatically change Skipped->Downloaded when updating
         # There was a bug report where this was causing infinite downloads if the album was
         # recent, but matched to less than 80%. It would go Downloaded->Skipped->Wanted->Downloaded->Skipped->Wanted->etc....
-        #else:
+        # else:
         #    if album['Status'] == "Skipped" or album['Status'] == "Downloaded":
         #        new_album_status = "Skipped"
         #    else:
@@ -375,7 +436,7 @@ def update_album_status(AlbumID=None):
         else:
             new_album_status = album['Status']
 
-        myDB.upsert("albums", {'Status'   : new_album_status}, {'AlbumID'   : album['AlbumID']})
+        myDB.upsert("albums", {'Status': new_album_status}, {'AlbumID': album['AlbumID']})
         if new_album_status != album['Status']:
             logger.info('Album %s changed to %s' % (album['AlbumTitle'], new_album_status))
     logger.info('Album status update complete')

headphones/lock.py

@@ -0,0 +1,88 @@
+"""
+Locking-related classes
+"""
+
+import time
+import threading
+import Queue
+
+import headphones.logger
+
+
+class TimedLock(object):
+    """
+    Enforce request rate limit if applicable. This uses the lock so there
+    is synchronized access to the API. When N threads enter this method, the
+    first will pass trough, since there there was no last request recorded.
+    The last request time will be set. Then, the second thread will unlock,
+    and see that the last request was X seconds ago. It will sleep
+    (request_limit - X) seconds, and then continue. Then the third one will
+    unblock, and so on. After all threads finish, the total time will at
+    least be (N * request_limit) seconds. If some request takes longer than
+    request_limit seconds, the next unblocked thread will wait less.
+    """
+
+    def __init__(self, minimum_delta=0):
+        """
+        Set up the lock
+        """
+        self.lock = threading.Lock()
+        self.last_used = 0
+        self.minimum_delta = minimum_delta
+        self.queue = Queue.Queue()
+
+    def __enter__(self):
+        """
+        Called when with lock: is invoked
+        """
+        self.lock.acquire()
+        delta = time.time() - self.last_used
+        sleep_amount = self.minimum_delta - delta
+        if sleep_amount >= 0:
+            # zero sleeps give the cpu a chance to task-switch
+            headphones.logger.debug('Sleeping %s (interval)', sleep_amount)
+            time.sleep(sleep_amount)
+        while not self.queue.empty():
+            try:
+                seconds = self.queue.get(False)
+                headphones.logger.debug('Sleeping %s (queued)', seconds)
+                time.sleep(seconds)
+            except Queue.Empty:
+                continue
+            self.queue.task_done()
+
+    def __exit__(self, type, value, traceback):
+        """
+        Called when exiting the with block.
+        """
+        self.last_used = time.time()
+        self.lock.release()
+
+    def snooze(self, seconds):
+        """
+        Asynchronously add time to the next request. Can be called outside
+        of the lock context, but it is possible for the next lock holder
+        to not check the queue until after something adds time to it.
+        """
+        # We use a queue so that we don't have to synchronize
+        # across threads and with or without locks
+        headphones.logger.info('Adding %s to queue', seconds)
+        self.queue.put(seconds)
+
+
+class FakeLock(object):
+    """
+    If no locking or request throttling is needed, use this
+    """
+
+    def __enter__(self):
+        """
+        Do nothing on enter
+        """
+        pass
+
+    def __exit__(self, type, value, traceback):
+        """
+        Do nothing on exit
+        """
+        pass

headphones/logger.py

@@ -13,30 +13,32 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import helpers
-
-from logutils.queue import QueueHandler, QueueListener
 from logging import handlers
-
 import multiprocessing
 import contextlib
-import headphones
 import threading
 import traceback
 import logging
+import errno
 import sys
+
 import os
+from headphones import helpers
+from logutils.queue import QueueHandler, QueueListener
+import headphones
+
 
 # These settings are for file logging only
 FILENAME = "headphones.log"
-MAX_SIZE = 1000000 # 1 MB
+MAX_SIZE = 1000000  # 1 MB
 MAX_FILES = 5
 
 # Headphones logger
 logger = logging.getLogger("headphones")
 
-# Global queue of multiprocessing logging
-queue = multiprocessing.Queue()
+# Global queue for multiprocessing logging
+queue = None
+
 
 class LogListHandler(logging.Handler):
     """
@@ -49,6 +51,7 @@ class LogListHandler(logging.Handler):
 
         headphones.LOG_LIST.insert(0, (helpers.now(), message, record.levelname, record.threadName))
 
+
 @contextlib.contextmanager
 def listener():
     """
@@ -56,13 +59,34 @@ def listener():
     To be used in a with statement in the main process, for multiprocessing.
     """
 
-    queue_listener = QueueListener(queue, *logger.handlers)
+    global queue
 
-    try:
-        queue_listener.start()
+    # Initialize queue if not already done
+    if queue is None:
+        try:
+            queue = multiprocessing.Queue()
+        except OSError as e:
+            queue = False
+
+            # Some machines don't have access to /dev/shm. See
+            # http://stackoverflow.com/questions/2009278 for more information.
+            if e.errno == errno.EACCES:
+                logger.warning("Multiprocess logging disabled, because "
+                               "current user cannot map shared memory. You won't see any" \
+                               "logging generated by the worker processed.")
+
+    # Multiprocess logging may be disabled.
+    if not queue:
         yield
-    finally:
-        queue_listener.stop()
+    else:
+        queue_listener = QueueListener(queue, *logger.handlers)
+
+        try:
+            queue_listener.start()
+            yield
+        finally:
+            queue_listener.stop()
+
 
 def initMultiprocessing():
     """
@@ -71,6 +95,11 @@ def initMultiprocessing():
     completely.
     """
 
+    # Multiprocess logging may be disabled.
+    if not queue:
+        return
+
+    # Remove all handlers and add the Queue handler as the only one.
     for handler in logger.handlers[:]:
         logger.removeHandler(handler)
 
@@ -82,7 +111,8 @@ def initMultiprocessing():
     # Change current thread name for log record
     threading.current_thread().name = multiprocessing.current_process().name
 
-def initLogger(console=False, verbose=False):
+
+def initLogger(console=False, log_dir=False, verbose=False):
     """
     Setup logging for Headphones. It uses the logger instance with the name
     'headphones'. Three log handlers are added:
@@ -91,7 +121,8 @@ def initLogger(console=False, verbose=False):
     * LogListHandler: for Web UI
     * StreamHandler: for console (if console)
 
-    Console logging is only enabled if console is set to True.
+    Console logging is only enabled if console is set to True. This method can
+    be invoked multiple times, during different stages of Headphones.
     """
 
     # Close and remove old handlers. This is required to reinit the loggers
@@ -109,25 +140,29 @@ def initLogger(console=False, verbose=False):
     logger.propagate = False
     logger.setLevel(logging.DEBUG if verbose else logging.INFO)
 
-    # Setup file logger
-    filename = os.path.join(headphones.LOG_DIR, FILENAME)
-
-    file_formatter = logging.Formatter('%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
-    file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE, backupCount=MAX_FILES)
-    file_handler.setLevel(logging.DEBUG)
-    file_handler.setFormatter(file_formatter)
-
-    logger.addHandler(file_handler)
-
     # Add list logger
     loglist_handler = LogListHandler()
     loglist_handler.setLevel(logging.DEBUG)
 
     logger.addHandler(loglist_handler)
 
+    # Setup file logger
+    if log_dir:
+        filename = os.path.join(log_dir, FILENAME)
+
+        file_formatter = logging.Formatter(
+            '%(asctime)s - %(levelname)-7s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
+        file_handler = handlers.RotatingFileHandler(filename, maxBytes=MAX_SIZE,
+                                                    backupCount=MAX_FILES)
+        file_handler.setLevel(logging.DEBUG)
+        file_handler.setFormatter(file_formatter)
+
+        logger.addHandler(file_handler)
+
     # Setup console logger
     if console:
-        console_formatter = logging.Formatter('%(asctime)s - %(levelname)s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
+        console_formatter = logging.Formatter(
+            '%(asctime)s - %(levelname)s :: %(threadName)s : %(message)s', '%d-%b-%Y %H:%M:%S')
         console_handler = logging.StreamHandler()
         console_handler.setFormatter(console_formatter)
         console_handler.setLevel(logging.DEBUG)
@@ -137,6 +172,7 @@ def initLogger(console=False, verbose=False):
     # Install exception hooks
     initHooks()
 
+
 def initHooks(global_exceptions=True, thread_exceptions=True, pass_original=True):
     """
     This method installs exception catching mechanisms. Any exception caught
@@ -180,15 +216,17 @@ def initHooks(global_exceptions=True, thread_exceptions=True, pass_original=True
                     raise
                 except:
                     excepthook(*sys.exc_info())
+
             self.run = new_run
 
         # Monkey patch the run() by monkey patching the __init__ method
         threading.Thread.__init__ = new_init
 
+
 # Expose logger methods
 info = logger.info
 warn = logger.warn
 error = logger.error
 debug = logger.debug
 warning = logger.warning
-exception = logger.exception
+exception = logger.exception

headphones/lyrics.py

@@ -13,17 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import re
 import htmlentitydefs
 
+import re
 from headphones import logger, request
 
-def getLyrics(artist, song):
 
-    params = {  "artist": artist.encode('utf-8'),
-                "song": song.encode('utf-8'),
-                "fmt": 'xml'
-                }
+def getLyrics(artist, song):
+    params = {"artist": artist.encode('utf-8'),
+              "song": song.encode('utf-8'),
+              "fmt": 'xml'
+              }
 
     url = 'http://lyrics.wikia.com/api.php'
     data = request.request_minidom(url, params=params)
@@ -45,10 +45,13 @@ def getLyrics(artist, song):
         logger.warn('Error fetching lyrics from: %s' % lyricsurl)
         return
 
-    m = re.compile('''<div class='lyricbox'><div class='rtMatcher'>.*?</div>(.*?)<!--''').search(lyricspage)
+    m = re.compile('''<div class='lyricbox'><div class='rtMatcher'>.*?</div>(.*?)<!--''').search(
+        lyricspage)
 
     if not m:
-        m = re.compile('''<div class='lyricbox'><span style="padding:1em"><a href="/Category:Instrumental" title="Instrumental">''').search(lyricspage)
+        m = re.compile(
+            '''<div class='lyricbox'><span style="padding:1em"><a href="/Category:Instrumental" title="Instrumental">''').search(
+            lyricspage)
         if m:
             return u'(Instrumental)'
         else:
@@ -60,17 +63,18 @@ def getLyrics(artist, song):
 
     return lyrics
 
+
 def convert_html_entities(s):
     matches = re.findall("&#\d+;", s)
     if len(matches) > 0:
         hits = set(matches)
         for hit in hits:
-                name = hit[2:-1]
-                try:
-                        entnum = int(name)
-                        s = s.replace(hit, unichr(entnum))
-                except ValueError:
-                        pass
+            name = hit[2:-1]
+            try:
+                entnum = int(name)
+                s = s.replace(hit, unichr(entnum))
+            except ValueError:
+                pass
 
     matches = re.findall("&\w+;", s)
     hits = set(matches)
@@ -79,7 +83,7 @@ def convert_html_entities(s):
         hits.remove(amp)
     for hit in hits:
         name = hit[1:-1]
-        if htmlentitydefs.name2codepoint.has_key(name):
-                s = s.replace(hit, unichr(htmlentitydefs.name2codepoint[name]))
+        if name in htmlentitydefs.name2codepoint:
+            s = s.replace(hit, unichr(htmlentitydefs.name2codepoint[name]))
     s = s.replace(amp, "&")
     return s

headphones/metacritic.py

@@ -0,0 +1,87 @@
+#  This file is part of Headphones.
+#
+#  Headphones is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  Headphones is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+
+from headphones import db, helpers, logger, request
+
+
+def update(artistid, artist_name, release_groups):
+    """ Pretty simple and crude function to find the artist page on metacritic,
+    then parse that page to get critic & user scores for albums"""
+
+    # First let's modify the artist name to fit the metacritic convention.
+    # We could just do a search, then take the top result, but at least this will
+    # cut down on api calls. If it's ineffective then we'll switch to search
+
+    replacements = {" & ": " ", ".": ""}
+    mc_artist_name = helpers.replace_all(artist_name.lower(), replacements)
+
+    mc_artist_name = mc_artist_name.replace(" ", "-")
+
+    headers = {
+        'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2243.2 Safari/537.36'}
+
+    url = "http://www.metacritic.com/person/" + mc_artist_name + "?filter-options=music&sort_options=date&num_items=100"
+
+    res = request.request_soup(url, headers=headers)
+
+    rows = None
+
+    try:
+        table = res.find("table", class_="credits person_credits")
+        rows = table.tbody.find_all('tr')
+    except:
+        logger.info("Unable to get metacritic scores for: %s" % artist_name)
+
+    myDB = db.DBConnection()
+    artist = myDB.action('SELECT * FROM artists WHERE ArtistID=?', [artistid]).fetchone()
+
+    score_list = []
+
+    # If we couldn't get anything from MetaCritic for whatever reason,
+    # let's try to load scores from the db
+    if not rows:
+        if artist['MetaCritic']:
+            score_list = json.loads(artist['MetaCritic'])
+        else:
+            return
+
+    # If we did get scores, let's update the db with them
+    else:
+        for row in rows:
+            title = row.a.string
+            scores = row.find_all("span")
+            critic_score = scores[0].string
+            user_score = scores[1].string
+            score_dict = {'title': title, 'critic_score': critic_score, 'user_score': user_score}
+            score_list.append(score_dict)
+
+        # Save scores to the database
+        controlValueDict = {"ArtistID": artistid}
+        newValueDict = {'MetaCritic': json.dumps(score_list)}
+        myDB.upsert("artists", newValueDict, controlValueDict)
+
+    for score in score_list:
+        title = score['title']
+        # Iterate through the release groups we got passed to see if we can find
+        # a match
+        for rg in release_groups:
+            if rg['title'].lower() == title.lower():
+                critic_score = score['critic_score']
+                user_score = score['user_score']
+                controlValueDict = {"AlbumID": rg['id']}
+                newValueDict = {'CriticScore': critic_score, 'UserScore': user_score}
+                myDB.upsert("albums", newValueDict, controlValueDict)

headphones/music_encoder.py

@@ -13,37 +13,39 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
-import headphones
-import shutil
 import time
+import shutil
+import subprocess
 import multiprocessing
 
-import subprocess
+import os
+import headphones
 from headphones import logger
 from beets.mediafile import MediaFile
 
+
 # xld
-if headphones.ENCODER == 'xld':
-    import getXldProfile
-    XLD = True
-else:
-    XLD = False
+import getXldProfile
+
 
 def encode(albumPath):
+    use_xld = headphones.CONFIG.ENCODER == 'xld'
 
     # Return if xld details not found
-    if XLD:
-        global xldProfile
-        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(headphones.XLDPROFILE)
+    if use_xld:
+        (xldProfile, xldFormat, xldBitrate) = getXldProfile.getXldProfile(
+            headphones.CONFIG.XLDPROFILE)
         if not xldFormat:
-            logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded', xldProfile)
+            logger.error('Details for xld profile \'%s\' not found, files will not be re-encoded',
+                         xldProfile)
             return None
+    else:
+        xldProfile = None
 
-    tempDirEncode=os.path.join(albumPath,"temp")
-    musicFiles=[]
-    musicFinalFiles=[]
-    musicTempFiles=[]
+    tempDirEncode = os.path.join(albumPath, "temp")
+    musicFiles = []
+    musicFinalFiles = []
+    musicTempFiles = []
     encoder = ""
 
     # Create temporary directory, but remove the old one first.
@@ -57,21 +59,24 @@ def encode(albumPath):
         logger.exception("Unable to create temporary directory")
         return None
 
-    for r,d,f in os.walk(albumPath):
+    for r, d, f in os.walk(albumPath):
         for music in f:
             if any(music.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
-                if not XLD:
-                    encoderFormat = headphones.ENCODEROUTPUTFORMAT.encode(headphones.SYS_ENCODING)
+                if not use_xld:
+                    encoderFormat = headphones.CONFIG.ENCODEROUTPUTFORMAT.encode(
+                        headphones.SYS_ENCODING)
                 else:
                     xldMusicFile = os.path.join(r, music)
                     xldInfoMusic = MediaFile(xldMusicFile)
                     encoderFormat = xldFormat
 
-                if (headphones.ENCODERLOSSLESS):
+                if headphones.CONFIG.ENCODERLOSSLESS:
                     ext = os.path.normpath(os.path.splitext(music)[1].lstrip(".")).lower()
-                    if not XLD and ext == 'flac' or XLD and (ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
+                    if not use_xld and ext == 'flac' or use_xld and (
+                            ext != xldFormat and (xldInfoMusic.bitrate / 1000 > 400)):
                         musicFiles.append(os.path.join(r, music))
-                        musicTemp = os.path.normpath(os.path.splitext(music)[0] + '.' + encoderFormat)
+                        musicTemp = os.path.normpath(
+                            os.path.splitext(music)[0] + '.' + encoderFormat)
                         musicTempFiles.append(os.path.join(tempDirEncode, musicTemp))
                     else:
                         logger.debug('%s is already encoded', music)
@@ -80,29 +85,29 @@ def encode(albumPath):
                     musicTemp = os.path.normpath(os.path.splitext(music)[0] + '.' + encoderFormat)
                     musicTempFiles.append(os.path.join(tempDirEncode, musicTemp))
 
-    if headphones.ENCODER_PATH:
-        encoder = headphones.ENCODER_PATH.encode(headphones.SYS_ENCODING)
+    if headphones.CONFIG.ENCODER_PATH:
+        encoder = headphones.CONFIG.ENCODER_PATH.encode(headphones.SYS_ENCODING)
     else:
-        if XLD:
+        if use_xld:
             encoder = os.path.join('/Applications', 'xld')
-        elif headphones.ENCODER =='lame':
+        elif headphones.CONFIG.ENCODER == 'lame':
             if headphones.SYS_PLATFORM == "win32":
-                ## NEED THE DEFAULT LAME INSTALL ON WIN!
+                # NEED THE DEFAULT LAME INSTALL ON WIN!
                 encoder = "C:/Program Files/lame/lame.exe"
             else:
-                encoder="lame"
-        elif headphones.ENCODER =='ffmpeg':
+                encoder = "lame"
+        elif headphones.CONFIG.ENCODER == 'ffmpeg':
             if headphones.SYS_PLATFORM == "win32":
                 encoder = "C:/Program Files/ffmpeg/bin/ffmpeg.exe"
             else:
-                encoder="ffmpeg"
-        elif headphones.ENCODER == 'libav':
+                encoder = "ffmpeg"
+        elif headphones.CONFIG.ENCODER == 'libav':
             if headphones.SYS_PLATFORM == "win32":
                 encoder = "C:/Program Files/libav/bin/avconv.exe"
             else:
-                encoder="avconv"
+                encoder = "avconv"
 
-    i=0
+    i = 0
     encoder_failed = False
     jobs = []
 
@@ -110,53 +115,60 @@ def encode(albumPath):
         infoMusic = MediaFile(music)
         encode = False
 
-        if XLD:
+        if use_xld:
             if xldBitrate and (infoMusic.bitrate / 1000 <= xldBitrate):
-                logger.info('%s has bitrate <= %skb, will not be re-encoded', music.decode(headphones.SYS_ENCODING, 'replace'), xldBitrate)
+                logger.info('%s has bitrate <= %skb, will not be re-encoded',
+                            music.decode(headphones.SYS_ENCODING, 'replace'), xldBitrate)
             else:
                 encode = True
-        elif headphones.ENCODER == 'lame':
-            if not any(music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + x) for x in ["mp3", "wav"]):
-                logger.warn('Lame cannot encode %s format for %s, use ffmpeg', os.path.splitext(music)[1], music)
+        elif headphones.CONFIG.ENCODER == 'lame':
+            if not any(
+                    music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + x) for x
+                    in ["mp3", "wav"]):
+                logger.warn('Lame cannot encode %s format for %s, use ffmpeg',
+                            os.path.splitext(music)[1], music)
             else:
-                if (music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (int(infoMusic.bitrate / 1000) <= headphones.BITRATE)):
-                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.BITRATE)
+                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.mp3') and (
+                    int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
+                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music,
+                                headphones.CONFIG.BITRATE)
                 else:
                     encode = True
         else:
-            if headphones.ENCODEROUTPUTFORMAT=='ogg':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                 if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.ogg'):
-                    logger.warn('Cannot re-encode .ogg %s', music.decode(headphones.SYS_ENCODING, 'replace'))
+                    logger.warn('Cannot re-encode .ogg %s',
+                                music.decode(headphones.SYS_ENCODING, 'replace'))
                 else:
                     encode = True
-            elif (headphones.ENCODEROUTPUTFORMAT=='mp3' or headphones.ENCODEROUTPUTFORMAT=='m4a'):
-                if (music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.'+headphones.ENCODEROUTPUTFORMAT) and (int(infoMusic.bitrate / 1000 ) <= headphones.BITRATE)):
-                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.BITRATE)
+            else:
+                if music.decode(headphones.SYS_ENCODING, 'replace').lower().endswith('.' + headphones.CONFIG.ENCODEROUTPUTFORMAT) and (int(infoMusic.bitrate / 1000) <= headphones.CONFIG.BITRATE):
+                    logger.info('%s has bitrate <= %skb, will not be re-encoded', music, headphones.CONFIG.BITRATE)
                 else:
                     encode = True
         # encode
         if encode:
-            job = (encoder, music, musicTempFiles[i], albumPath)
+            job = (encoder, music, musicTempFiles[i], albumPath, xldProfile)
             jobs.append(job)
         else:
             musicFiles[i] = None
             musicTempFiles[i] = None
 
-        i=i+1
+        i = i + 1
 
     # Encode music files
     if len(jobs) > 0:
         processes = 1
 
         # Use multicore if enabled
-        if headphones.ENCODER_MULTICORE:
-            if headphones.ENCODER_MULTICORE_COUNT == 0:
+        if headphones.CONFIG.ENCODER_MULTICORE:
+            if headphones.CONFIG.ENCODER_MULTICORE_COUNT == 0:
                 processes = multiprocessing.cpu_count()
             else:
-                processes = headphones.ENCODER_MULTICORE_COUNT
+                processes = headphones.CONFIG.ENCODER_MULTICORE_COUNT
 
             logger.debug("Multi-core encoding enabled, spawning %d processes",
-                processes)
+                         processes)
 
         # Use multiprocessing only if it's worth the overhead. and if it is
         # enabled. If not, then use the old fashioned way.
@@ -186,7 +198,8 @@ def encode(albumPath):
         for dest in musicTempFiles:
             if not os.path.exists(dest):
                 encoder_failed = True
-                logger.error("Encoded file '%s' does not exist in the destination temp directory", dest)
+                logger.error("Encoded file '%s' does not exist in the destination temp directory",
+                             dest)
 
     # No errors, move from temp to parent
     if not encoder_failed and musicTempFiles:
@@ -194,14 +207,14 @@ def encode(albumPath):
         for dest in musicTempFiles:
             if os.path.exists(dest):
                 source = musicFiles[i]
-                if headphones.DELETE_LOSSLESS_FILES:
+                if headphones.CONFIG.DELETE_LOSSLESS_FILES:
                     os.remove(source)
                 check_dest = os.path.join(albumPath, os.path.split(dest)[1])
                 if os.path.exists(check_dest):
                     os.remove(check_dest)
                 try:
                     shutil.move(dest, albumPath)
-                except Exception, e:
+                except Exception as e:
                     logger.error('Could not move %s to %s: %s', dest, albumPath, e)
                     encoder_failed = True
                     break
@@ -212,11 +225,13 @@ def encode(albumPath):
 
     # Return with error if any encoding errors
     if encoder_failed:
-        logger.error("One or more files failed to encode. Ensure you have the latest version of %s installed.", headphones.ENCODER)
+        logger.error(
+            "One or more files failed to encode. Ensure you have the latest version of %s installed.",
+            headphones.CONFIG.ENCODER)
         return None
 
     time.sleep(1)
-    for r,d,f in os.walk(albumPath):
+    for r, d, f in os.walk(albumPath):
         for music in f:
             if any(music.lower().endswith('.' + x.lower()) for x in headphones.MEDIA_FORMATS):
                 musicFinalFiles.append(os.path.join(r, music))
@@ -226,6 +241,7 @@ def encode(albumPath):
 
     return musicFinalFiles
 
+
 def command_map(args):
     """
     Wrapper for the '[multiprocessing.]map()' method, to unpack the arguments
@@ -239,11 +255,12 @@ def command_map(args):
     # Start encoding
     try:
         return command(*args)
-    except Exception as e:
+    except Exception:
         logger.exception("Encoder raised an exception.")
         return False
 
-def command(encoder, musicSource, musicDest, albumPath):
+
+def command(encoder, musicSource, musicDest, albumPath, xldProfile):
     """
     Encode a given music file with a certain encoder. Returns True on success,
     or False otherwise.
@@ -252,8 +269,7 @@ def command(encoder, musicSource, musicDest, albumPath):
     startMusicTime = time.time()
     cmd = []
 
-    # XLD
-    if XLD:
+    if xldProfile:
         xldDestDir = os.path.split(musicDest)[0]
         cmd = [encoder]
         cmd.extend([musicSource])
@@ -263,17 +279,18 @@ def command(encoder, musicSource, musicDest, albumPath):
         cmd.extend([xldDestDir])
 
     # Lame
-    elif headphones.ENCODER == 'lame':
+    elif headphones.CONFIG.ENCODER == 'lame':
         cmd = [encoder]
         opts = []
-        if not headphones.ADVANCEDENCODER:
+        if not headphones.CONFIG.ADVANCEDENCODER:
             opts.extend(['-h'])
-            if headphones.ENCODERVBRCBR=='cbr':
-                opts.extend(['--resample', str(headphones.SAMPLINGFREQUENCY), '-b', str(headphones.BITRATE)])
-            elif headphones.ENCODERVBRCBR=='vbr':
-                opts.extend(['-v', str(headphones.ENCODERQUALITY)])
+            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
+                opts.extend(['--resample', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-b',
+                             str(headphones.CONFIG.BITRATE)])
+            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
+                opts.extend(['-v', str(headphones.CONFIG.ENCODERQUALITY)])
         else:
-            advanced = (headphones.ADVANCEDENCODER.split())
+            advanced = (headphones.CONFIG.ADVANCEDENCODER.split())
             for tok in advanced:
                 opts.extend([tok.encode(headphones.SYS_ENCODING)])
         opts.extend([musicSource])
@@ -281,42 +298,44 @@ def command(encoder, musicSource, musicDest, albumPath):
         cmd.extend(opts)
 
     # FFmpeg
-    elif headphones.ENCODER == 'ffmpeg':
+    elif headphones.CONFIG.ENCODER == 'ffmpeg':
         cmd = [encoder, '-i', musicSource]
         opts = []
-        if not headphones.ADVANCEDENCODER:
-            if headphones.ENCODEROUTPUTFORMAT=='ogg':
+        if not headphones.CONFIG.ADVANCEDENCODER:
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                 opts.extend(['-acodec', 'libvorbis'])
-            if headphones.ENCODEROUTPUTFORMAT=='m4a':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
                 opts.extend(['-strict', 'experimental'])
-            if headphones.ENCODERVBRCBR=='cbr':
-                opts.extend(['-ar', str(headphones.SAMPLINGFREQUENCY), '-ab', str(headphones.BITRATE) + 'k'])
-            elif headphones.ENCODERVBRCBR=='vbr':
-                opts.extend(['-aq', str(headphones.ENCODERQUALITY)])
+            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
+                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab',
+                             str(headphones.CONFIG.BITRATE) + 'k'])
+            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
+                opts.extend(['-aq', str(headphones.CONFIG.ENCODERQUALITY)])
             opts.extend(['-y', '-ac', '2', '-vn'])
         else:
-            advanced = (headphones.ADVANCEDENCODER.split())
+            advanced = (headphones.CONFIG.ADVANCEDENCODER.split())
             for tok in advanced:
                 opts.extend([tok.encode(headphones.SYS_ENCODING)])
         opts.extend([musicDest])
         cmd.extend(opts)
 
     # Libav
-    elif headphones.ENCODER == "libav":
+    elif headphones.CONFIG.ENCODER == "libav":
         cmd = [encoder, '-i', musicSource]
         opts = []
-        if not headphones.ADVANCEDENCODER:
-            if headphones.ENCODEROUTPUTFORMAT=='ogg':
+        if not headphones.CONFIG.ADVANCEDENCODER:
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'ogg':
                 opts.extend(['-acodec', 'libvorbis'])
-            if headphones.ENCODEROUTPUTFORMAT=='m4a':
+            if headphones.CONFIG.ENCODEROUTPUTFORMAT == 'm4a':
                 opts.extend(['-strict', 'experimental'])
-            if headphones.ENCODERVBRCBR=='cbr':
-                opts.extend(['-ar', str(headphones.SAMPLINGFREQUENCY), '-ab', str(headphones.BITRATE) + 'k'])
-            elif headphones.ENCODERVBRCBR=='vbr':
-                opts.extend(['-aq', str(headphones.ENCODERQUALITY)])
+            if headphones.CONFIG.ENCODERVBRCBR == 'cbr':
+                opts.extend(['-ar', str(headphones.CONFIG.SAMPLINGFREQUENCY), '-ab',
+                             str(headphones.CONFIG.BITRATE) + 'k'])
+            elif headphones.CONFIG.ENCODERVBRCBR == 'vbr':
+                opts.extend(['-aq', str(headphones.CONFIG.ENCODERQUALITY)])
             opts.extend(['-y', '-ac', '2', '-vn'])
         else:
-            advanced = (headphones.ADVANCEDENCODER.split())
+            advanced = (headphones.CONFIG.ADVANCEDENCODER.split())
             for tok in advanced:
                 opts.extend([tok.encode(headphones.SYS_ENCODING)])
         opts.extend([musicDest])
@@ -337,17 +356,18 @@ def command(encoder, musicSource, musicDest, albumPath):
     logger.debug(subprocess.list2cmdline(cmd))
 
     process = subprocess.Popen(cmd, startupinfo=startupinfo,
-        stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
-        stderr=subprocess.PIPE)
-    stdout, stderr = process.communicate(headphones.ENCODER)
+                               stdin=open(os.devnull, 'rb'), stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE)
+    stdout, stderr = process.communicate(headphones.CONFIG.ENCODER)
 
     # Error if return code not zero
     if process.returncode:
-        logger.error('Encoding failed for %s' % (musicSource.decode(headphones.SYS_ENCODING, 'replace')))
+        logger.error(
+            'Encoding failed for %s' % (musicSource.decode(headphones.SYS_ENCODING, 'replace')))
         out = stdout if stdout else stderr
         out = out.decode(headphones.SYS_ENCODING, 'replace')
         outlast2lines = '\n'.join(out.splitlines()[-2:])
-        logger.error('%s error details: %s' % (headphones.ENCODER, outlast2lines))
+        logger.error('%s error details: %s' % (headphones.CONFIG.ENCODER, outlast2lines))
         out = out.rstrip("\n")
         logger.debug(out)
         encoded = False
@@ -357,10 +377,11 @@ def command(encoder, musicSource, musicDest, albumPath):
 
     return encoded
 
+
 def getTimeEncode(start):
-    seconds =int(time.time()-start)
+    seconds = int(time.time() - start)
     hours = seconds / 3600
-    seconds -= 3600*hours
+    seconds -= 3600 * hours
     minutes = seconds / 60
-    seconds -= 60*minutes
-    return "%02d:%02d:%02d" % (hours, minutes, seconds)
+    seconds -= 60 * minutes
+    return "%02d:%02d:%02d" % (hours, minutes, seconds)

headphones/notifiers.py

@@ -13,41 +13,36 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger, helpers, common, request
-
-from xml.dom import minidom
-from httplib import HTTPSConnection
 from urllib import urlencode
-from pynma import pynma
-
-import base64
-import cherrypy
 import urllib
-import urllib2
-import headphones
-import os.path
 import subprocess
-import gntp.notifier
-import time
 import json
+from email.mime.text import MIMEText
+import smtplib
+import email.utils
+from httplib import HTTPSConnection
+from urlparse import parse_qsl
+import urllib2
 
+import os.path
+from headphones import logger, helpers, common, request
+from pynma import pynma
+import cherrypy
+import headphones
+import gntp.notifier
 import oauth2 as oauth
 import pythontwitter as twitter
 
-try:
-    from urlparse import parse_qsl
-except ImportError:
-    from cgi import parse_qsl
 
 class GROWL(object):
     """
-    Growl notifications, for OS X
+    Growl notifications, for OS X.
     """
 
     def __init__(self):
-        self.enabled = headphones.GROWL_ENABLED
-        self.host = headphones.GROWL_HOST
-        self.password = headphones.GROWL_PASSWORD
+        self.enabled = headphones.CONFIG.GROWL_ENABLED
+        self.host = headphones.CONFIG.GROWL_HOST
+        self.password = headphones.CONFIG.GROWL_PASSWORD
 
     def conf(self, options):
         return cherrypy.config['config'].get('Growl', options)
@@ -84,14 +79,18 @@ class GROWL(object):
         try:
             growl.register()
         except gntp.notifier.errors.NetworkError:
-            logger.info(u'Growl notification failed: network error')
+            logger.warning(u'Growl notification failed: network error')
             return
         except gntp.notifier.errors.AuthError:
-            logger.info(u'Growl notification failed: authentication error')
+            logger.warning(u'Growl notification failed: authentication error')
             return
 
+        # Fix message
+        message = message.encode(headphones.SYS_ENCODING, "replace")
+
         # Send it, including an image
-        image_file = os.path.join(str(headphones.PROG_DIR), 'data/images/headphoneslogo.png')
+        image_file = os.path.join(str(headphones.PROG_DIR),
+                                  "data/images/headphoneslogo.png")
 
         with open(image_file, 'rb') as f:
             image = f.read()
@@ -104,13 +103,13 @@ class GROWL(object):
                 icon=image
             )
         except gntp.notifier.errors.NetworkError:
-            logger.info(u'Growl notification failed: network error')
+            logger.warning(u'Growl notification failed: network error')
             return
 
         logger.info(u"Growl notifications sent.")
 
     def updateLibrary(self):
-        #For uniformity reasons not removed
+        # For uniformity reasons not removed
         return
 
     def test(self, host, password):
@@ -120,50 +119,51 @@ class GROWL(object):
 
         self.notify('ZOMG Lazors Pewpewpew!', 'Test Message')
 
+
 class PROWL(object):
     """
     Prowl notifications.
     """
 
     def __init__(self):
-        self.enabled = headphones.PROWL_ENABLED
-        self.keys = headphones.PROWL_KEYS
-        self.priority = headphones.PROWL_PRIORITY
+        self.enabled = headphones.CONFIG.PROWL_ENABLED
+        self.keys = headphones.CONFIG.PROWL_KEYS
+        self.priority = headphones.CONFIG.PROWL_PRIORITY
 
     def conf(self, options):
         return cherrypy.config['config'].get('Prowl', options)
 
     def notify(self, message, event):
-        if not headphones.PROWL_ENABLED:
+        if not headphones.CONFIG.PROWL_ENABLED:
             return
 
         http_handler = HTTPSConnection("api.prowlapp.com")
 
-        data = {'apikey': headphones.PROWL_KEYS,
+        data = {'apikey': headphones.CONFIG.PROWL_KEYS,
                 'application': 'Headphones',
                 'event': event,
                 'description': message.encode("utf-8"),
-                'priority': headphones.PROWL_PRIORITY }
+                'priority': headphones.CONFIG.PROWL_PRIORITY}
 
         http_handler.request("POST",
-                                "/publicapi/add",
-                                headers = {'Content-type': "application/x-www-form-urlencoded"},
-                                body = urlencode(data))
+                             "/publicapi/add",
+                             headers={'Content-type': "application/x-www-form-urlencoded"},
+                             body=urlencode(data))
         response = http_handler.getresponse()
         request_status = response.status
 
         if request_status == 200:
-                logger.info(u"Prowl notifications sent.")
-                return True
+            logger.info(u"Prowl notifications sent.")
+            return True
         elif request_status == 401:
-                logger.info(u"Prowl auth failed: %s" % response.reason)
-                return False
+            logger.info(u"Prowl auth failed: %s" % response.reason)
+            return False
         else:
-                logger.info(u"Prowl notification failed.")
-                return False
+            logger.info(u"Prowl notification failed.")
+            return False
 
     def updateLibrary(self):
-        #For uniformity reasons not removed
+        # For uniformity reasons not removed
         return
 
     def test(self, keys, priority):
@@ -173,17 +173,17 @@ class PROWL(object):
 
         self.notify('ZOMG Lazors Pewpewpew!', 'Test Message')
 
+
 class MPC(object):
     """
     MPC library update
     """
 
     def __init__(self):
-
         pass
 
-    def notify( self ):
-        subprocess.call( ["mpc", "update"] )
+    def notify(self):
+        subprocess.call(["mpc", "update"])
 
 
 class XBMC(object):
@@ -193,9 +193,9 @@ class XBMC(object):
 
     def __init__(self):
 
-        self.hosts = headphones.XBMC_HOST
-        self.username = headphones.XBMC_USERNAME
-        self.password = headphones.XBMC_PASSWORD
+        self.hosts = headphones.CONFIG.XBMC_HOST
+        self.username = headphones.CONFIG.XBMC_USERNAME
+        self.password = headphones.CONFIG.XBMC_PASSWORD
 
     def _sendhttp(self, host, command):
         url_command = urllib.urlencode(command)
@@ -212,9 +212,11 @@ class XBMC(object):
         url = host + '/jsonrpc'
 
         if self.password:
-            response = request.request_json(url, method="post", data=json.dumps(data), headers=headers, auth=(self.username, self.password))
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers, auth=(self.username, self.password))
         else:
-            response = request.request_json(url, method="post", data=json.dumps(data), headers=headers)
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers)
 
         if response:
             return response[0]['result']
@@ -226,7 +228,7 @@ class XBMC(object):
         hosts = [x.strip() for x in self.hosts.split(',')]
 
         for host in hosts:
-            logger.info('Sending library update command to XBMC @ '+host)
+            logger.info('Sending library update command to XBMC @ ' + host)
             request = self._sendjson(host, 'AudioLibrary.Scan')
 
             if not request:
@@ -238,20 +240,24 @@ class XBMC(object):
 
         header = "Headphones"
         message = "%s - %s added to your library" % (artist, album)
-        time = "3000" # in ms
+        time = "3000"  # in ms
 
         for host in hosts:
-            logger.info('Sending notification command to XMBC @ '+host)
+            logger.info('Sending notification command to XMBC @ ' + host)
             try:
-                version = self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})['version']['major']
+                version = \
+                self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})[
+                    'version']['major']
 
-                if version < 12: #Eden
+                if version < 12:  # Eden
                     notification = header + "," + message + "," + time + "," + albumartpath
-                    notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification('+notification+')'}
+                    notifycommand = {'command': 'ExecBuiltIn',
+                                     'parameter': 'Notification(' + notification + ')'}
                     request = self._sendhttp(host, notifycommand)
 
-                else: #Frodo
-                    params = {'title':header, 'message': message, 'displaytime': int(time), 'image': albumartpath}
+                else:  # Frodo
+                    params = {'title': header, 'message': message, 'displaytime': int(time),
+                              'image': albumartpath}
                     request = self._sendjson(host, 'GUI.ShowNotification', params)
 
                 if not request:
@@ -260,25 +266,26 @@ class XBMC(object):
             except Exception:
                 logger.error('Error sending notification request to XBMC')
 
+
 class LMS(object):
     """
     Class for updating a Logitech Media Server
     """
 
     def __init__(self):
-        self.hosts = headphones.LMS_HOST
+        self.hosts = headphones.CONFIG.LMS_HOST
 
     def _sendjson(self, host):
-        data = {'id': 1, 'method': 'slim.request', 'params': ["",["rescan"]]}
+        data = {'id': 1, 'method': 'slim.request', 'params': ["", ["rescan"]]}
         data = json.JSONEncoder().encode(data)
 
         content = {'Content-Type': 'application/json'}
 
-        req = urllib2.Request(host+'/jsonrpc.js', data, content)
+        req = urllib2.Request(host + '/jsonrpc.js', data, content)
 
         try:
             handle = urllib2.urlopen(req)
-        except Exception, e:
+        except Exception as e:
             logger.warn('Error opening LMS url: %s' % e)
             return
 
@@ -288,54 +295,55 @@ class LMS(object):
             return response['result']
         except:
             logger.warn('LMS returned error: %s' % response['error'])
-            return
+            return response['error']
 
     def update(self):
 
         hosts = [x.strip() for x in self.hosts.split(',')]
 
         for host in hosts:
-            logger.info('Sending library rescan command to LMS @ '+host)
+            logger.info('Sending library rescan command to LMS @ ' + host)
             request = self._sendjson(host)
 
-            if not request:
+            if request:
                 logger.warn('Error sending rescan request to LMS')
 
+
 class Plex(object):
     def __init__(self):
 
-        self.server_hosts = headphones.PLEX_SERVER_HOST
-        self.client_hosts = headphones.PLEX_CLIENT_HOST
-        self.username = headphones.PLEX_USERNAME
-        self.password = headphones.PLEX_PASSWORD
+        self.server_hosts = headphones.CONFIG.PLEX_SERVER_HOST
+        self.client_hosts = headphones.CONFIG.PLEX_CLIENT_HOST
+        self.username = headphones.CONFIG.PLEX_USERNAME
+        self.password = headphones.CONFIG.PLEX_PASSWORD
+        self.token = headphones.CONFIG.PLEX_TOKEN
 
     def _sendhttp(self, host, command):
 
-        username = self.username
-        password = self.password
+        url = host + '/xbmcCmds/xbmcHttp/?' + command
 
-        url_command = urllib.urlencode(command)
-
-        url = host + '/xbmcCmds/xbmcHttp/?' + url_command
-
-        req = urllib2.Request(url)
-
-        if password:
-            base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
-            req.add_header("Authorization", "Basic %s" % base64string)
-
-        logger.info('Plex url: %s' % url)
-
-        try:
-            handle = urllib2.urlopen(req)
-        except Exception, e:
-            logger.warn('Error opening Plex url: %s' % e)
-            return
-
-        response = handle.read().decode(headphones.SYS_ENCODING)
+        if self.password:
+            response = request.request_response(url, auth=(self.username, self.password))
+        else:
+            response = request.request_response(url)
 
         return response
 
+    def _sendjson(self, host, method, params={}):
+        data = [{'id': 0, 'jsonrpc': '2.0', 'method': method, 'params': params}]
+        headers = {'Content-Type': 'application/json'}
+        url = host + '/jsonrpc'
+
+        if self.password:
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers, auth=(self.username, self.password))
+        else:
+            response = request.request_json(url, method="post", data=json.dumps(data),
+                                            headers=headers)
+
+        if response:
+            return response[0]['result']
+
     def update(self):
 
         # From what I read you can't update the music library on a per directory or per path basis
@@ -344,15 +352,17 @@ class Plex(object):
         hosts = [x.strip() for x in self.server_hosts.split(',')]
 
         for host in hosts:
-            logger.info('Sending library update command to Plex Media Server@ '+host)
+            logger.info('Sending library update command to Plex Media Server@ ' + host)
             url = "%s/library/sections" % host
-            try:
-                xml_sections = minidom.parse(urllib.urlopen(url))
-            except IOError, e:
-                logger.warn("Error while trying to contact Plex Media Server: %s" % e)
-                return False
+            if self.token:
+                params = {'X-Plex-Token': self.token}
+            else:
+                params = False
+
+            r = request.request_minidom(url, params=params)
+
+            sections = r.getElementsByTagName('Directory')
 
-            sections = xml_sections.getElementsByTagName('Directory')
             if not sections:
                 logger.info(u"Plex Media Server not running on: " + host)
                 return False
@@ -360,11 +370,7 @@ class Plex(object):
             for s in sections:
                 if s.getAttribute('type') == "artist":
                     url = "%s/library/sections/%s/refresh" % (host, s.getAttribute('key'))
-                    try:
-                        urllib.urlopen(url)
-                    except Exception, e:
-                        logger.warn("Error updating library section for Plex Media Server: %s" % e)
-                        return False
+                    request.request_response(url, params=params)
 
     def notify(self, artist, album, albumartpath):
 
@@ -372,26 +378,38 @@ class Plex(object):
 
         header = "Headphones"
         message = "%s - %s added to your library" % (artist, album)
-        time = "3000" # in ms
+        time = "3000"  # in ms
 
         for host in hosts:
-            logger.info('Sending notification command to Plex Media Server @ '+host)
+            logger.info('Sending notification command to Plex client @ ' + host)
             try:
-                notification = header + "," + message + "," + time + "," + albumartpath
-                notifycommand = {'command': 'ExecBuiltIn', 'parameter': 'Notification('+notification+')'}
-                request = self._sendhttp(host, notifycommand)
+                version = \
+                self._sendjson(host, 'Application.GetProperties', {'properties': ['version']})[
+                    'version']['major']
+
+                if version < 12:  # Eden
+                    notification = header + "," + message + "," + time + "," + albumartpath
+                    notifycommand = {'command': 'ExecBuiltIn',
+                                     'parameter': 'Notification(' + notification + ')'}
+                    request = self._sendhttp(host, notifycommand)
+
+                else:  # Frodo
+                    params = {'title': header, 'message': message, 'displaytime': int(time),
+                              'image': albumartpath}
+                    request = self._sendjson(host, 'GUI.ShowNotification', params)
 
                 if not request:
                     raise Exception
 
-            except:
-                logger.warn('Error sending notification request to Plex Media Server')
+            except Exception:
+                logger.error('Error sending notification request to Plex client @ ' + host)
+
 
 class NMA(object):
     def notify(self, artist=None, album=None, snatched=None):
         title = 'Headphones'
-        api = headphones.NMA_APIKEY
-        nma_priority = headphones.NMA_PRIORITY
+        api = headphones.CONFIG.NMA_APIKEY
+        nma_priority = headphones.CONFIG.NMA_PRIORITY
 
         logger.debug(u"NMA title: " + title)
         logger.debug(u"NMA API: " + api)
@@ -413,7 +431,8 @@ class NMA(object):
         keys = api.split(',')
         p.addkey(keys)
 
-        if len(keys) > 1: batch = True
+        if len(keys) > 1:
+            batch = True
 
         response = p.push(title, event, message, priority=nma_priority, batch_mode=batch)
 
@@ -423,66 +442,44 @@ class NMA(object):
         else:
             return True
 
+
 class PUSHBULLET(object):
-
     def __init__(self):
-        self.apikey = headphones.PUSHBULLET_APIKEY
-        self.deviceid = headphones.PUSHBULLET_DEVICEID
+        self.apikey = headphones.CONFIG.PUSHBULLET_APIKEY
+        self.deviceid = headphones.CONFIG.PUSHBULLET_DEVICEID
 
-    def conf(self, options):
-        return cherrypy.config['config'].get('PUSHBULLET', options)
-
-    def notify(self, message, event):
-        if not headphones.PUSHBULLET_ENABLED:
+    def notify(self, message, status):
+        if not headphones.CONFIG.PUSHBULLET_ENABLED:
             return
 
-        http_handler = HTTPSConnection("api.pushbullet.com")
+        url = "https://api.pushbullet.com/v2/pushes"
 
-        data = {'device_iden': headphones.PUSHBULLET_DEVICEID,
-                'type': "note",
+        data = {'type': "note",
                 'title': "Headphones",
-                'body': message.encode("utf-8") }
+                'body': message + ': ' + status}
 
-        http_handler.request("POST",
-                                "/api/pushes",
-                                headers = {'Content-type': "application/x-www-form-urlencoded",
-                                            'Authorization' : 'Basic %s' % base64.b64encode(headphones.PUSHBULLET_APIKEY + ":") },
-                                body = urlencode(data))
-        response = http_handler.getresponse()
-        request_status = response.status
-        logger.debug(u"PushBullet response status: %r" % request_status)
-        logger.debug(u"PushBullet response headers: %r" % response.getheaders())
-        logger.debug(u"PushBullet response body: %r" % response.read())
+        if self.deviceid:
+            data['device_iden'] = self.deviceid
 
-        if request_status == 200:
-                logger.info(u"PushBullet notifications sent.")
-                return True
-        elif request_status >= 400 and request_status < 500:
-                logger.info(u"PushBullet request failed: %s" % response.reason)
-                return False
+        headers = {'Content-type': "application/json",
+                   'Authorization': 'Bearer ' + headphones.CONFIG.PUSHBULLET_APIKEY}
+
+        response = request.request_json(url, method="post", headers=headers, data=json.dumps(data))
+
+        if response:
+            logger.info(u"PushBullet notifications sent.")
+            return True
         else:
-                logger.info(u"PushBullet notification failed serverside.")
-                return False
+            logger.info(u"PushBullet notification failed.")
+            return False
 
-    def updateLibrary(self):
-        #For uniformity reasons not removed
-        return
-
-    def test(self, apikey, deviceid):
-
-        self.enabled = True
-        self.apikey = apikey
-        self.deviceid = deviceid
-
-        self.notify('Main Screen Activate', 'Test Message')
 
 class PUSHALOT(object):
-
     def notify(self, message, event):
-        if not headphones.PUSHALOT_ENABLED:
+        if not headphones.CONFIG.PUSHALOT_ENABLED:
             return
 
-        pushalot_authorizationtoken = headphones.PUSHALOT_APIKEY
+        pushalot_authorizationtoken = headphones.CONFIG.PUSHALOT_APIKEY
 
         logger.debug(u"Pushalot event: " + event)
         logger.debug(u"Pushalot message: " + message)
@@ -492,12 +489,12 @@ class PUSHALOT(object):
 
         data = {'AuthorizationToken': pushalot_authorizationtoken,
                 'Title': event.encode('utf-8'),
-                'Body': message.encode("utf-8") }
+                'Body': message.encode("utf-8")}
 
         http_handler.request("POST",
-                                "/api/sendmessage",
-                                headers = {'Content-type': "application/x-www-form-urlencoded"},
-                                body = urlencode(data))
+                             "/api/sendmessage",
+                             headers={'Content-type': "application/x-www-form-urlencoded"},
+                             body=urlencode(data))
         response = http_handler.getresponse()
         request_status = response.status
 
@@ -506,14 +503,15 @@ class PUSHALOT(object):
         logger.debug(u"Pushalot response body: %r" % response.read())
 
         if request_status == 200:
-                logger.info(u"Pushalot notifications sent.")
-                return True
+            logger.info(u"Pushalot notifications sent.")
+            return True
         elif request_status == 410:
-                logger.info(u"Pushalot auth failed: %s" % response.reason)
-                return False
+            logger.info(u"Pushalot auth failed: %s" % response.reason)
+            return False
         else:
-                logger.info(u"Pushalot notification failed.")
-                return False
+            logger.info(u"Pushalot notification failed.")
+            return False
+
 
 class Synoindex(object):
     def __init__(self, util_loc='/usr/syno/bin/synoindex'):
@@ -526,7 +524,8 @@ class Synoindex(object):
         path = os.path.abspath(path)
 
         if not self.util_exists():
-            logger.warn("Error sending notification: synoindex utility not found at %s" % self.util_loc)
+            logger.warn(
+                "Error sending notification: synoindex utility not found at %s" % self.util_loc)
             return
 
         if os.path.isfile(path):
@@ -534,15 +533,17 @@ class Synoindex(object):
         elif os.path.isdir(path):
             cmd_arg = '-A'
         else:
-            logger.warn("Error sending notification: Path passed to synoindex was not a file or folder.")
+            logger.warn(
+                "Error sending notification: Path passed to synoindex was not a file or folder.")
             return
 
         cmd = [self.util_loc, cmd_arg, path]
         logger.info("Calling synoindex command: %s" % str(cmd))
         try:
-            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=headphones.PROG_DIR)
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                                 cwd=headphones.PROG_DIR)
             out, error = p.communicate()
-            #synoindex never returns any codes other than '0', highly irritating
+            # synoindex never returns any codes other than '0', highly irritating
         except OSError, e:
             logger.warn("Error sending notification: %s" % str(e))
 
@@ -551,15 +552,15 @@ class Synoindex(object):
             for path in path_list:
                 self.notify(path)
 
+
 class PUSHOVER(object):
-
     def __init__(self):
-        self.enabled = headphones.PUSHOVER_ENABLED
-        self.keys = headphones.PUSHOVER_KEYS
-        self.priority = headphones.PUSHOVER_PRIORITY
+        self.enabled = headphones.CONFIG.PUSHOVER_ENABLED
+        self.keys = headphones.CONFIG.PUSHOVER_KEYS
+        self.priority = headphones.CONFIG.PUSHOVER_PRIORITY
 
-        if headphones.PUSHOVER_APITOKEN:
-            self.application_token = headphones.PUSHOVER_APITOKEN
+        if headphones.CONFIG.PUSHOVER_APITOKEN:
+            self.application_token = headphones.CONFIG.PUSHOVER_APITOKEN
         else:
             self.application_token = "LdPCoy0dqC21ktsbEyAVCcwvQiVlsz"
 
@@ -567,39 +568,30 @@ class PUSHOVER(object):
         return cherrypy.config['config'].get('Pushover', options)
 
     def notify(self, message, event):
-        if not headphones.PUSHOVER_ENABLED:
+        if not headphones.CONFIG.PUSHOVER_ENABLED:
             return
 
-        http_handler = HTTPSConnection("api.pushover.net")
+        url = "https://api.pushover.net/1/messages.json"
 
         data = {'token': self.application_token,
-                'user': headphones.PUSHOVER_KEYS,
+                'user': headphones.CONFIG.PUSHOVER_KEYS,
                 'title': event,
                 'message': message.encode("utf-8"),
-                'priority': headphones.PUSHOVER_PRIORITY }
+                'priority': headphones.CONFIG.PUSHOVER_PRIORITY}
 
-        http_handler.request("POST",
-                                "/1/messages.json",
-                                headers = {'Content-type': "application/x-www-form-urlencoded"},
-                                body = urlencode(data))
-        response = http_handler.getresponse()
-        request_status = response.status
-        logger.debug(u"Pushover response status: %r" % request_status)
-        logger.debug(u"Pushover response headers: %r" % response.getheaders())
-        logger.debug(u"Pushover response body: %r" % response.read())
+        headers = {'Content-type': "application/x-www-form-urlencoded"}
 
-        if request_status == 200:
-                logger.info(u"Pushover notifications sent.")
-                return True
-        elif request_status >= 400 and request_status < 500:
-                logger.info(u"Pushover request failed: %s" % response.reason)
-                return False
+        response = request.request_response(url, method="POST", headers=headers, data=data)
+
+        if response:
+            logger.info(u"Pushover notifications sent.")
+            return True
         else:
-                logger.info(u"Pushover notification failed.")
-                return False
+            logger.error(u"Pushover notification failed.")
+            return False
 
     def updateLibrary(self):
-        #For uniformity reasons not removed
+        # For uniformity reasons not removed
         return
 
     def test(self, keys, priority):
@@ -609,33 +601,35 @@ class PUSHOVER(object):
 
         self.notify('Main Screen Activate', 'Test Message')
 
-class TwitterNotifier(object):
 
+class TwitterNotifier(object):
     REQUEST_TOKEN_URL = 'https://api.twitter.com/oauth/request_token'
-    ACCESS_TOKEN_URL  = 'https://api.twitter.com/oauth/access_token'
+    ACCESS_TOKEN_URL = 'https://api.twitter.com/oauth/access_token'
     AUTHORIZATION_URL = 'https://api.twitter.com/oauth/authorize'
-    SIGNIN_URL        = 'https://api.twitter.com/oauth/authenticate'
+    SIGNIN_URL = 'https://api.twitter.com/oauth/authenticate'
 
     def __init__(self):
         self.consumer_key = "oYKnp2ddX5gbARjqX8ZAAg"
         self.consumer_secret = "A4Xkw9i5SjHbTk7XT8zzOPqivhj9MmRDR9Qn95YA9sk"
 
     def notify_snatch(self, title):
-        if headphones.TWITTER_ONSNATCH:
-            self._notifyTwitter(common.notifyStrings[common.NOTIFY_SNATCH]+': '+title+' at '+helpers.now())
+        if headphones.CONFIG.TWITTER_ONSNATCH:
+            self._notifyTwitter(
+                common.notifyStrings[common.NOTIFY_SNATCH] + ': ' + title + ' at ' + helpers.now())
 
     def notify_download(self, title):
-        if headphones.TWITTER_ENABLED:
-            self._notifyTwitter(common.notifyStrings[common.NOTIFY_DOWNLOAD]+': '+title+' at '+helpers.now())
+        if headphones.CONFIG.TWITTER_ENABLED:
+            self._notifyTwitter(common.notifyStrings[
+                                    common.NOTIFY_DOWNLOAD] + ': ' + title + ' at ' + helpers.now())
 
     def test_notify(self):
-        return self._notifyTwitter("This is a test notification from Headphones at "+helpers.now(), force=True)
+        return self._notifyTwitter(
+            "This is a test notification from Headphones at " + helpers.now(), force=True)
 
     def _get_authorization(self):
 
-        signature_method_hmac_sha1 = oauth.SignatureMethod_HMAC_SHA1() #@UnusedVariable
-        oauth_consumer             = oauth.Consumer(key=self.consumer_key, secret=self.consumer_secret)
-        oauth_client               = oauth.Client(oauth_consumer)
+        oauth_consumer = oauth.Consumer(key=self.consumer_key, secret=self.consumer_secret)
+        oauth_client = oauth.Client(oauth_consumer)
 
         logger.info('Requesting temp token from Twitter')
 
@@ -646,95 +640,106 @@ class TwitterNotifier(object):
         else:
             request_token = dict(parse_qsl(content))
 
-            headphones.TWITTER_USERNAME = request_token['oauth_token']
-            headphones.TWITTER_PASSWORD = request_token['oauth_token_secret']
+            headphones.CONFIG.TWITTER_USERNAME = request_token['oauth_token']
+            headphones.CONFIG.TWITTER_PASSWORD = request_token['oauth_token_secret']
 
-            return self.AUTHORIZATION_URL+"?oauth_token="+ request_token['oauth_token']
+            return self.AUTHORIZATION_URL + "?oauth_token=" + request_token['oauth_token']
 
     def _get_credentials(self, key):
         request_token = {}
 
-        request_token['oauth_token'] = headphones.TWITTER_USERNAME
-        request_token['oauth_token_secret'] = headphones.TWITTER_PASSWORD
+        request_token['oauth_token'] = headphones.CONFIG.TWITTER_USERNAME
+        request_token['oauth_token_secret'] = headphones.CONFIG.TWITTER_PASSWORD
         request_token['oauth_callback_confirmed'] = 'true'
 
         token = oauth.Token(request_token['oauth_token'], request_token['oauth_token_secret'])
         token.set_verifier(key)
 
-        logger.info('Generating and signing request for an access token using key '+key)
+        logger.info('Generating and signing request for an access token using key ' + key)
 
-        signature_method_hmac_sha1 = oauth.SignatureMethod_HMAC_SHA1() #@UnusedVariable
-        oauth_consumer             = oauth.Consumer(key=self.consumer_key, secret=self.consumer_secret)
-        logger.info('oauth_consumer: '+str(oauth_consumer))
-        oauth_client  = oauth.Client(oauth_consumer, token)
-        logger.info('oauth_client: '+str(oauth_client))
-        resp, content = oauth_client.request(self.ACCESS_TOKEN_URL, method='POST', body='oauth_verifier=%s' % key)
-        logger.info('resp, content: '+str(resp)+','+str(content))
+        oauth_consumer = oauth.Consumer(key=self.consumer_key, secret=self.consumer_secret)
+        logger.info('oauth_consumer: ' + str(oauth_consumer))
+        oauth_client = oauth.Client(oauth_consumer, token)
+        logger.info('oauth_client: ' + str(oauth_client))
+        resp, content = oauth_client.request(self.ACCESS_TOKEN_URL, method='POST',
+                                             body='oauth_verifier=%s' % key)
+        logger.info('resp, content: ' + str(resp) + ',' + str(content))
 
-        access_token  = dict(parse_qsl(content))
-        logger.info('access_token: '+str(access_token))
+        access_token = dict(parse_qsl(content))
+        logger.info('access_token: ' + str(access_token))
 
-        logger.info('resp[status] = '+str(resp['status']))
+        logger.info('resp[status] = ' + str(resp['status']))
         if resp['status'] != '200':
-            logger.info('The request for a token with did not succeed: '+str(resp['status']), logger.ERROR)
+            logger.info('The request for a token with did not succeed: ' + str(resp['status']),
+                        logger.ERROR)
             return False
         else:
             logger.info('Your Twitter Access Token key: %s' % access_token['oauth_token'])
             logger.info('Access Token secret: %s' % access_token['oauth_token_secret'])
-            headphones.TWITTER_USERNAME = access_token['oauth_token']
-            headphones.TWITTER_PASSWORD = access_token['oauth_token_secret']
+            headphones.CONFIG.TWITTER_USERNAME = access_token['oauth_token']
+            headphones.CONFIG.TWITTER_PASSWORD = access_token['oauth_token_secret']
             return True
 
-
     def _send_tweet(self, message=None):
 
-        username=self.consumer_key
-        password=self.consumer_secret
-        access_token_key=headphones.TWITTER_USERNAME
-        access_token_secret=headphones.TWITTER_PASSWORD
+        username = self.consumer_key
+        password = self.consumer_secret
+        access_token_key = headphones.CONFIG.TWITTER_USERNAME
+        access_token_secret = headphones.CONFIG.TWITTER_PASSWORD
 
-        logger.info(u"Sending tweet: "+message)
+        logger.info(u"Sending tweet: " + message)
 
         api = twitter.Api(username, password, access_token_key, access_token_secret)
 
         try:
             api.PostUpdate(message)
-        except Exception, e:
+        except Exception as e:
             logger.info(u"Error Sending Tweet: %s" % e)
             return False
 
         return True
 
     def _notifyTwitter(self, message='', force=False):
-        prefix = headphones.TWITTER_PREFIX
+        prefix = headphones.CONFIG.TWITTER_PREFIX
 
-        if not headphones.TWITTER_ENABLED and not force:
+        if not headphones.CONFIG.TWITTER_ENABLED and not force:
             return False
 
-        return self._send_tweet(prefix+": "+message)
+        return self._send_tweet(prefix + ": " + message)
+
 
 class OSX_NOTIFY(object):
-
     def __init__(self):
         try:
             self.objc = __import__("objc")
+            self.AppKit = __import__("AppKit")
         except:
+            logger.warn('OS X Notification: Cannot import objc or AppKit')
             return False
 
     def swizzle(self, cls, SEL, func):
-        old_IMP = cls.instanceMethodForSelector_(SEL)
+        old_IMP = getattr(cls, SEL, None)
+        if old_IMP is None:
+            old_IMP = cls.instanceMethodForSelector_(SEL)
+
         def wrapper(self, *args, **kwargs):
             return func(self, old_IMP, *args, **kwargs)
-        new_IMP = self.objc.selector(wrapper, selector=old_IMP.selector,
-            signature=old_IMP.signature)
-        self.objc.classAddMethod(cls, SEL, new_IMP)
 
-    def notify(self, title, subtitle=None, text=None, sound=True):
+        new_IMP = self.objc.selector(
+            wrapper,
+            selector=old_IMP.selector,
+            signature=old_IMP.signature
+        )
+        self.objc.classAddMethod(cls, SEL.encode(), new_IMP)
+
+    def notify(self, title, subtitle=None, text=None, sound=True, image=None):
 
         try:
-            self.swizzle(self.objc.lookUpClass('NSBundle'),
-                b'bundleIdentifier',
-                self.swizzled_bundleIdentifier)
+            self.swizzle(
+                self.objc.lookUpClass('NSBundle'),
+                'bundleIdentifier',
+                self.swizzled_bundleIdentifier
+            )
 
             NSUserNotification = self.objc.lookUpClass('NSUserNotification')
             NSUserNotificationCenter = self.objc.lookUpClass('NSUserNotificationCenter')
@@ -753,6 +758,10 @@ class OSX_NOTIFY(object):
                 notification.setInformativeText_(text)
             if sound:
                 notification.setSoundName_("NSUserNotificationDefaultSoundName")
+            if image:
+                source_img = self.AppKit.NSImage.alloc().initByReferencingFile_(image)
+                notification.setContentImage_(source_img)
+                # notification.set_identityImage_(source_img)
             notification.setHasActionButton_(False)
 
             notification_center = NSUserNotificationCenter.defaultUserNotificationCenter()
@@ -761,15 +770,15 @@ class OSX_NOTIFY(object):
             del pool
             return True
 
-        except Exception, e:
+        except Exception as e:
             logger.warn('Error sending OS X Notification: %s' % e)
             return False
 
     def swizzled_bundleIdentifier(self, original, swizzled):
         return 'ade.headphones.osxnotify'
 
-class BOXCAR(object):
 
+class BOXCAR(object):
     def __init__(self):
         self.url = 'https://new.boxcar.io/api/notifications'
 
@@ -779,11 +788,11 @@ class BOXCAR(object):
                 message += '<br></br><a href="http://musicbrainz.org/release-group/%s">MusicBrainz</a>' % rgid
 
             data = urllib.urlencode({
-                'user_credentials': headphones.BOXCAR_TOKEN,
+                'user_credentials': headphones.CONFIG.BOXCAR_TOKEN,
                 'notification[title]': title.encode('utf-8'),
                 'notification[long_message]': message.encode('utf-8'),
                 'notification[sound]': "done"
-                })
+            })
 
             req = urllib2.Request(self.url)
             handle = urllib2.urlopen(req, data)
@@ -794,12 +803,12 @@ class BOXCAR(object):
             logger.warn('Error sending Boxcar2 Notification: %s' % e)
             return False
 
-class SubSonicNotifier(object):
 
+class SubSonicNotifier(object):
     def __init__(self):
-        self.host = headphones.SUBSONIC_HOST
-        self.username = headphones.SUBSONIC_USERNAME
-        self.password = headphones.SUBSONIC_PASSWORD
+        self.host = headphones.CONFIG.SUBSONIC_HOST
+        self.username = headphones.CONFIG.SUBSONIC_USERNAME
+        self.password = headphones.CONFIG.SUBSONIC_PASSWORD
 
     def notify(self, albumpaths):
         # Correct URL
@@ -811,4 +820,39 @@ class SubSonicNotifier(object):
 
         # Invoke request
         request.request_response(self.host + "musicFolderSettings.view?scanNow",
-            auth=(self.username, self.password))
+                                 auth=(self.username, self.password))
+
+
+class Email(object):
+    def notify(self, subject, message):
+
+        message = MIMEText(message, 'plain', "utf-8")
+        message['Subject'] = subject
+        message['From'] = email.utils.formataddr(('Headphones', headphones.CONFIG.EMAIL_FROM))
+        message['To'] = headphones.CONFIG.EMAIL_TO
+
+        try:
+            if headphones.CONFIG.EMAIL_SSL:
+                mailserver = smtplib.SMTP_SSL(headphones.CONFIG.EMAIL_SMTP_SERVER,
+                                              headphones.CONFIG.EMAIL_SMTP_PORT)
+            else:
+                mailserver = smtplib.SMTP(headphones.CONFIG.EMAIL_SMTP_SERVER,
+                                          headphones.CONFIG.EMAIL_SMTP_PORT)
+
+            if headphones.CONFIG.EMAIL_TLS:
+                mailserver.starttls()
+
+            mailserver.ehlo()
+
+            if headphones.CONFIG.EMAIL_SMTP_USER:
+                mailserver.login(headphones.CONFIG.EMAIL_SMTP_USER,
+                                 headphones.CONFIG.EMAIL_SMTP_PASSWORD)
+
+            mailserver.sendmail(headphones.CONFIG.EMAIL_FROM, headphones.CONFIG.EMAIL_TO,
+                                message.as_string())
+            mailserver.quit()
+            return True
+
+        except Exception, e:
+            logger.warn('Error sending Email: %s' % e)
+            return False

headphones/nzbget.py

@@ -19,51 +19,49 @@
 # along with Sick Beard.  If not, see <http://www.gnu.org/licenses/>.
 
 
-
-import httplib
-import datetime
-
-import headphones
-
 from base64 import standard_b64encode
+import httplib
 import xmlrpclib
 
-#from headphones.providers.generic import GenericProvider
-
+import headphones
 from headphones import logger
 
+
 def sendNZB(nzb):
-
     addToTop = False
-    nzbgetXMLrpc = "%(username)s:%(password)s@%(host)s/xmlrpc"
+    nzbgetXMLrpc = "%(protocol)s://%(username)s:%(password)s@%(host)s/xmlrpc"
 
-    if headphones.NZBGET_HOST == None:
+    if not headphones.CONFIG.NZBGET_HOST:
         logger.error(u"No NZBget host found in configuration. Please configure it.")
         return False
 
-    if headphones.NZBGET_HOST.startswith('https://'):
-        nzbgetXMLrpc = 'https://' + nzbgetXMLrpc
-        headphones.NZBGET_HOST.replace('https://','',1)
+    if headphones.CONFIG.NZBGET_HOST.startswith('https://'):
+        protocol = 'https'
+        host = headphones.CONFIG.NZBGET_HOST.replace('https://', '', 1)
     else:
-        nzbgetXMLrpc = 'http://' + nzbgetXMLrpc
-        headphones.NZBGET_HOST.replace('http://','',1)
+        protocol = 'http'
+        host = headphones.CONFIG.NZBGET_HOST.replace('http://', '', 1)
 
-
-    url = nzbgetXMLrpc % {"host": headphones.NZBGET_HOST, "username": headphones.NZBGET_USERNAME, "password": headphones.NZBGET_PASSWORD}
+    url = nzbgetXMLrpc % {"protocol": protocol, "host": host,
+                          "username": headphones.CONFIG.NZBGET_USERNAME,
+                          "password": headphones.CONFIG.NZBGET_PASSWORD}
 
     nzbGetRPC = xmlrpclib.ServerProxy(url)
     try:
-        if nzbGetRPC.writelog("INFO", "headphones connected to drop of %s any moment now." % (nzb.name + ".nzb")):
+        if nzbGetRPC.writelog("INFO", "headphones connected to drop of %s any moment now." % (
+            nzb.name + ".nzb")):
             logger.debug(u"Successfully connected to NZBget")
         else:
-            logger.info(u"Successfully connected to NZBget, but unable to send a message" % (nzb.name + ".nzb"))
+            logger.info(u"Successfully connected to NZBget, but unable to send a message" % (
+            nzb.name + ".nzb"))
 
     except httplib.socket.error:
-        logger.error(u"Please check your NZBget host and port (if it is running). NZBget is not responding to this combination")
+        logger.error(
+            u"Please check your NZBget host and port (if it is running). NZBget is not responding to this combination")
         return False
 
     except xmlrpclib.ProtocolError, e:
-        if (e.errmsg == "Unauthorized"):
+        if e.errmsg == "Unauthorized":
             logger.error(u"NZBget password is incorrect.")
         else:
             logger.error(u"Protocol Error: " + e.errmsg)
@@ -86,35 +84,50 @@ def sendNZB(nzb):
         nzbget_version = int(nzbget_version_str[:nzbget_version_str.find(".")])
         if nzbget_version == 0:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, addToTop, nzbcontent64)
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb",
+                                                 headphones.CONFIG.NZBGET_CATEGORY, addToTop,
+                                                 nzbcontent64)
             else:
-                if nzb.resultType == "nzb":
-                    genProvider = GenericProvider("")
-                    data = genProvider.getURL(nzb.url)
-                    if (data == None):
-                        return False
-                    nzbcontent64 = standard_b64encode(data)
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, addToTop, nzbcontent64)
+                # from headphones.common.providers.generic import GenericProvider
+                # if nzb.resultType == "nzb":
+                #     genProvider = GenericProvider("")
+                #     data = genProvider.getURL(nzb.url)
+                #     if (data is None):
+                #         return False
+                #     nzbcontent64 = standard_b64encode(data)
+                # nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.CONFIG.NZBGET_CATEGORY, addToTop, nzbcontent64)
+                return False
         elif nzbget_version == 12:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb",
+                                                 headphones.CONFIG.NZBGET_CATEGORY,
+                                                 headphones.CONFIG.NZBGET_PRIORITY, False,
                                                  nzbcontent64, False, dupekey, dupescore, "score")
             else:
-                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb",
+                                                    headphones.CONFIG.NZBGET_CATEGORY,
+                                                    headphones.CONFIG.NZBGET_PRIORITY, False,
                                                     nzb.url, False, dupekey, dupescore, "score")
         # v13+ has a new combined append method that accepts both (url and content)
         # also the return value has changed from boolean to integer
         # (Positive number representing NZBID of the queue item. 0 and negative numbers represent error codes.)
         elif nzbget_version >= 13:
-            nzbget_result = True if nzbGetRPC.append(nzb.name + ".nzb", nzbcontent64 if nzbcontent64 is not None else nzb.url,
-                                                     headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False, False, dupekey, dupescore,
+            nzbget_result = True if nzbGetRPC.append(nzb.name + ".nzb",
+                                                     nzbcontent64 if nzbcontent64 is not None else nzb.url,
+                                                     headphones.CONFIG.NZBGET_CATEGORY,
+                                                     headphones.CONFIG.NZBGET_PRIORITY, False,
+                                                     False, dupekey, dupescore,
                                                      "score") > 0 else False
         else:
             if nzbcontent64 is not None:
-                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.append(nzb.name + ".nzb",
+                                                 headphones.CONFIG.NZBGET_CATEGORY,
+                                                 headphones.CONFIG.NZBGET_PRIORITY, False,
                                                  nzbcontent64)
             else:
-                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb", headphones.NZBGET_CATEGORY, headphones.NZBGET_PRIORITY, False,
+                nzbget_result = nzbGetRPC.appendurl(nzb.name + ".nzb",
+                                                    headphones.CONFIG.NZBGET_CATEGORY,
+                                                    headphones.CONFIG.NZBGET_PRIORITY, False,
                                                     nzb.url)
 
         if nzbget_result:
@@ -124,5 +137,6 @@ def sendNZB(nzb):
             logger.error(u"NZBget could not add %s to the queue" % (nzb.name + ".nzb"))
             return False
     except:
-        logger.error(u"Connect Error to NZBget: could not add %s to the queue" % (nzb.name + ".nzb"))
+        logger.error(
+            u"Connect Error to NZBget: could not add %s to the queue" % (nzb.name + ".nzb"))
         return False

headphones/request.py

@@ -13,21 +13,38 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger
-
 from xml.dom import minidom
-from bs4 import BeautifulSoup
+import collections
 
+import sys
+from bs4 import BeautifulSoup
 import requests
+from headphones import logger
 import feedparser
 import headphones
+import headphones.lock
+
+
+# Disable SSL certificate warnings. We have our own handling
+requests.packages.urllib3.disable_warnings()
+
+# Dictionary with last request times, for rate limiting.
+last_requests = collections.defaultdict(int)
+fake_lock = headphones.lock.FakeLock()
+
 
 def request_response(url, method="get", auto_raise=True,
-    whitelist_status_code=None, **kwargs):
+                     whitelist_status_code=None, lock=fake_lock, **kwargs):
     """
-    Convenient wrapper for `requests.get', which will capture the exceptions and
-    log them. On success, the Response object is returned. In case of a
+    Convenient wrapper for `requests.get', which will capture the exceptions
+    and log them. On success, the Response object is returned. In case of a
     exception, None is returned.
+
+    Additionally, there is support for rate limiting. To use this feature,
+    supply a tuple of (lock, request_limit). The lock is used to make sure no
+    other request with the same lock is executed. The request limit is the
+    minimal time between two requests (and so 1/request_limit is the number of
+    requests per seconds).
     """
 
     # Convert whitelist_status_code to a list if needed
@@ -36,16 +53,26 @@ def request_response(url, method="get", auto_raise=True,
 
     # Disable verification of SSL certificates if requested. Note: this could
     # pose a security issue!
-    kwargs["verify"] = headphones.VERIFY_SSL_CERT
+    kwargs["verify"] = bool(headphones.CONFIG.VERIFY_SSL_CERT)
 
-    # Map method to the request.XXX method. This is a simple hack, but it allows
-    # requests to apply more magic per method. See lib/requests/api.py.
+    #This fix is put in place for systems with broken SSL (like QNAP)
+    if not headphones.CONFIG.VERIFY_SSL_CERT and sys.version_info >= (2, 7, 9):
+        try:
+            import ssl
+            ssl._create_default_https_context = ssl._create_unverified_context
+        except:
+            pass
+
+    # Map method to the request.XXX method. This is a simple hack, but it
+    # allows requests to apply more magic per method. See lib/requests/api.py.
     request_method = getattr(requests, method.lower())
 
     try:
-        # Request the URL
-        logger.debug("Requesting URL via %s method: %s", method.upper(), url)
-        response = request_method(url, **kwargs)
+        # Request URL and wait for response
+        with lock:
+            logger.debug(
+                "Requesting URL via %s method: %s", method.upper(), url)
+            response = request_method(url, **kwargs)
 
         # If status code != OK, then raise exception, except if the status code
         # is white listed.
@@ -54,19 +81,34 @@ def request_response(url, method="get", auto_raise=True,
                 try:
                     response.raise_for_status()
                 except:
-                    logger.debug("Response status code %d is not white " \
-                         "listed, raised exception", response.status_code)
+                    logger.debug(
+                        "Response status code %d is not white "
+                        "listed, raised exception", response.status_code)
                     raise
         elif auto_raise:
             response.raise_for_status()
 
         return response
+    except requests.exceptions.SSLError as e:
+        if kwargs["verify"]:
+            logger.error(
+                "Unable to connect to remote host because of a SSL error. "
+                "It is likely that your system cannot verify the validity"
+                "of the certificate. The remote certificate is either "
+                "self-signed, or the remote server uses SNI. See the wiki for "
+                "more information on this topic.")
+        else:
+            logger.error(
+                "SSL error raised during connection, with certificate "
+                "verification turned off: %s", e)
     except requests.ConnectionError:
-        logger.error("Unable to connect to remote host. Check if the remote " \
+        logger.error(
+            "Unable to connect to remote host. Check if the remote "
             "host is up and running.")
     except requests.Timeout:
-        logger.error("Request timed out. The remote host did not respeond " \
-            "timely.")
+        logger.error(
+            "Request timed out. The remote host did not respond in a timely "
+            "manner.")
     except requests.HTTPError as e:
         if e.response is not None:
             if e.response.status_code >= 500:
@@ -77,7 +119,8 @@ def request_response(url, method="get", auto_raise=True,
                 # I don't think we will end up here, but for completeness
                 cause = "unknown"
 
-            logger.error("Request raise HTTP error with status code %d (%s).",
+            logger.error(
+                "Request raise HTTP error with status code %d (%s).",
                 e.response.status_code, cause)
 
             # Debug response
@@ -88,6 +131,7 @@ def request_response(url, method="get", auto_raise=True,
     except requests.RequestException as e:
         logger.error("Request raised exception: %s", e)
 
+
 def request_soup(url, **kwargs):
     """
     Wrapper for `request_response', which will return a BeatifulSoup object if
@@ -100,6 +144,7 @@ def request_soup(url, **kwargs):
     if response is not None:
         return BeautifulSoup(response.content, parser)
 
+
 def request_minidom(url, **kwargs):
     """
     Wrapper for `request_response', which will return a Minidom object if no
@@ -111,13 +156,14 @@ def request_minidom(url, **kwargs):
     if response is not None:
         return minidom.parseString(response.content)
 
+
 def request_json(url, **kwargs):
     """
     Wrapper for `request_response', which will decode the response as JSON
     object and return the result, if no exceptions are raised.
 
-    As an option, a validator callback can be given, which should return True if
-    the result is valid.
+    As an option, a validator callback can be given, which should return True
+    if the result is valid.
     """
 
     validator = kwargs.pop("validator", None)
@@ -138,6 +184,7 @@ def request_json(url, **kwargs):
             if headphones.VERBOSE:
                 server_message(response)
 
+
 def request_content(url, **kwargs):
     """
     Wrapper for `request_response', which will return the raw content.
@@ -148,6 +195,7 @@ def request_content(url, **kwargs):
     if response is not None:
         return response.content
 
+
 def request_feed(url, **kwargs):
     """
     Wrapper for `request_response', which will return a feed object.
@@ -158,6 +206,7 @@ def request_feed(url, **kwargs):
     if response is not None:
         return feedparser.parse(response.content)
 
+
 def server_message(response):
     """
     Extract server message from response and log in to logger with DEBUG level.
@@ -170,7 +219,8 @@ def server_message(response):
     message = None
 
     # First attempt is to 'read' the response as HTML
-    if "text/html" in response.headers.get("content-type"):
+    if response.headers.get("content-type") and \
+                    "text/html" in response.headers.get("content-type"):
         try:
             soup = BeautifulSoup(response.content, "html5lib")
         except Exception:
@@ -179,9 +229,9 @@ def server_message(response):
         # Find body and cleanup common tags to grab content, which probably
         # contains the message.
         message = soup.find("body")
+        elements = ("header", "script", "footer", "nav", "input", "textarea")
 
-        for element in ["header", "script", "footer", "nav", "input",
-            "textarea"]:
+        for element in elements:
 
             for tag in soup.find_all(element):
                 tag.replaceWith("")
@@ -198,4 +248,4 @@ def server_message(response):
         if len(message) > 150:
             message = message[:150] + "..."
 
-        logger.debug("Server responded with message: %s", message)
+        logger.debug("Server responded with message: %s", message)

headphones/rutracker.py

@@ -0,0 +1,226 @@
+#!/usr/bin/env python
+
+import urllib
+import time
+from urlparse import urlparse
+
+import re
+import requests as requests
+from bs4 import BeautifulSoup
+import headphones
+from headphones import logger
+
+
+class Rutracker(object):
+    def __init__(self):
+        self.session = requests.session()
+        self.timeout = 60
+        self.loggedin = False
+        self.maxsize = 0
+        self.search_referer = 'http://rutracker.org/forum/tracker.php'
+
+    def logged_in(self):
+        return self.loggedin
+
+    def still_logged_in(self, html):
+        if not html or "action=\"http://login.rutracker.org/forum/login.php\">" in html:
+            return False
+        else:
+            return True
+
+    def login(self):
+        """
+        Logs in user
+        """
+
+        loginpage = 'http://login.rutracker.org/forum/login.php'
+        post_params = {
+            'login_username': headphones.CONFIG.RUTRACKER_USER,
+            'login_password': headphones.CONFIG.RUTRACKER_PASSWORD,
+            'login': b'\xc2\xf5\xee\xe4'  # '%C2%F5%EE%E4'
+        }
+
+        logger.info("Attempting to log in to rutracker...")
+
+        try:
+            r = self.session.post(loginpage, data=post_params, timeout=self.timeout)
+            # try again
+            if 'bb_data' not in r.cookies.keys():
+                time.sleep(10)
+                r = self.session.post(loginpage, data=post_params, timeout=self.timeout)
+            if r.status_code != 200:
+                logger.error("rutracker login returned status code %s" % r.status_code)
+                self.loggedin = False
+            else:
+                if 'bb_data' in r.cookies.keys():
+                    self.loggedin = True
+                    logger.info("Successfully logged in to rutracker")
+                else:
+                    logger.error(
+                        "Could not login to rutracker, credentials maybe incorrect, site is down or too many attempts. Try again later")
+                    self.loggedin = False
+            return self.loggedin
+        except Exception as e:
+            logger.error("Unknown error logging in to rutracker: %s" % e)
+            self.loggedin = False
+            return self.loggedin
+
+    def searchurl(self, artist, album, year, format):
+        """
+        Return the search url
+        """
+
+        # Build search url
+        searchterm = ''
+        if artist != 'Various Artists':
+            searchterm = artist
+            searchterm = searchterm + ' '
+        searchterm = searchterm + album
+        searchterm = searchterm + ' '
+        searchterm = searchterm + year
+
+        if format == 'lossless':
+            format = '+lossless'
+            self.maxsize = 10000000000
+        elif format == 'lossless+mp3':
+            format = '+lossless||mp3||aac'
+            self.maxsize = 10000000000
+        else:
+            format = '+mp3||aac'
+            self.maxsize = 300000000
+
+        # sort by size, descending.
+        sort = '&o=7&s=2'
+
+        searchurl = "%s?nm=%s%s%s" % (self.search_referer, urllib.quote(searchterm), format, sort)
+
+        logger.info("Searching rutracker using term: %s", searchterm)
+
+        return searchurl
+
+    def search(self, searchurl):
+        """
+        Parse the search results and return valid torrent list
+        """
+
+        try:
+            headers = {'Referer': self.search_referer}
+            r = self.session.get(url=searchurl, headers=headers, timeout=self.timeout)
+
+            soup = BeautifulSoup(r.content, 'html5lib')
+
+            # Debug
+            # logger.debug (soup.prettify())
+
+            # Check if still logged in
+            if not self.still_logged_in(soup):
+                self.login()
+                r = self.session.get(url=searchurl, timeout=self.timeout)
+                soup = BeautifulSoup(r.content, 'html5lib')
+                if not self.still_logged_in(soup):
+                    logger.error("Error getting rutracker data")
+                    return None
+
+            # Process
+            rulist = []
+            i = soup.find('table', id='tor-tbl')
+            if not i:
+                logger.info("No valid results found from rutracker")
+                return None
+            minimumseeders = int(headphones.CONFIG.NUMBEROFSEEDERS) - 1
+
+            for item in zip(i.find_all(class_='hl-tags'), i.find_all(class_='dl-stub'),
+                            i.find_all(class_='seedmed')):
+                title = item[0].get_text()
+                url = item[1].get('href')
+                size_formatted = item[1].get_text()[:-2]
+                seeds = item[2].get_text()
+                size_parts = size_formatted.split()
+                size = float(size_parts[0])
+
+                if size_parts[1] == 'KB':
+                    size *= 1024
+                if size_parts[1] == 'MB':
+                    size *= 1024 ** 2
+                if size_parts[1] == 'GB':
+                    size *= 1024 ** 3
+                if size_parts[1] == 'TB':
+                    size *= 1024 ** 4
+
+                if size < self.maxsize and minimumseeders < int(seeds):
+                    logger.info('Found %s. Size: %s' % (title, size_formatted))
+                    # Torrent topic page
+                    torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])[
+                        't']
+                    topicurl = 'http://rutracker.org/forum/viewtopic.php?t=' + torrent_id
+                    rulist.append((title, size, topicurl, 'rutracker.org', 'torrent', True))
+                else:
+                    logger.info(
+                        "%s is larger than the maxsize or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i)" % (
+                        title, size, int(seeds)))
+
+            if not rulist:
+                logger.info("No valid results found from rutracker")
+
+            return rulist
+
+        except Exception as e:
+            logger.error("An unknown error occurred in the rutracker parser: %s" % e)
+            return None
+
+    def get_torrent_data(self, url):
+        """
+        return the .torrent data
+        """
+
+        torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
+        downloadurl = 'http://dl.rutracker.org/forum/dl.php?t=' + torrent_id
+        cookie = {'bb_dl': torrent_id}
+        try:
+            headers = {'Referer': url}
+            r = self.session.post(url=downloadurl, cookies=cookie, headers=headers,
+                                  timeout=self.timeout)
+            return r.content
+        except Exception as e:
+            logger.error('Error getting torrent: %s', e)
+            return False
+
+    # TODO get this working in utorrent.py
+    def utorrent_add_file(self, data):
+
+        host = headphones.CONFIG.UTORRENT_HOST
+        if not host.startswith('http'):
+            host = 'http://' + host
+        if host.endswith('/'):
+            host = host[:-1]
+        if host.endswith('/gui'):
+            host = host[:-4]
+
+        base_url = host
+
+        url = base_url + '/gui/'
+        self.session.auth = (
+        headphones.CONFIG.UTORRENT_USERNAME, headphones.CONFIG.UTORRENT_PASSWORD)
+
+        try:
+            r = self.session.get(url + 'token.html')
+        except Exception as e:
+            logger.error('Error getting token: %s', e)
+            return
+
+        if r.status_code == 401:
+            logger.debug('Error reaching utorrent')
+            return
+
+        regex = re.search(r'.+>([^<]+)</div></html>', r.text)
+        if regex is None:
+            logger.debug('Error reading token')
+            return
+
+        self.session.params = {'token': regex.group(1)}
+        files = {'torrent_file': ("", data)}
+
+        try:
+            self.session.post(url, params={'action': 'add-file'}, files=files)
+        except Exception as e:
+            logger.exception('Error adding file to utorrent %s', e)

headphones/sab.py

@@ -13,43 +13,58 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-#####################################
-## Stolen from Sick-Beard's sab.py ##
-#####################################
+###################################
+# Stolen from Sick-Beard's sab.py #
+###################################
 
-import MultipartPostHandler
-import headphones
-import datetime
 import cookielib
-import urllib2
-import httplib
-import urllib
-import ast
 
+import headphones
 from headphones.common import USER_AGENT
-from headphones import logger
-from headphones import notifiers, helpers
+from headphones import logger, helpers, request
+
+
+def sab_api_call(request_type=None, params={}, **kwargs):
+    if not headphones.CONFIG.SAB_HOST.startswith('http'):
+        headphones.CONFIG.SAB_HOST = 'http://' + headphones.CONFIG.SAB_HOST
+
+    if headphones.CONFIG.SAB_HOST.endswith('/'):
+        headphones.CONFIG.SAB_HOST = headphones.CONFIG.SAB_HOST[
+                                     0:len(headphones.CONFIG.SAB_HOST) - 1]
+
+    url = headphones.CONFIG.SAB_HOST + "/" + "api?"
+
+    if headphones.CONFIG.SAB_USERNAME:
+        params['ma_username'] = headphones.CONFIG.SAB_USERNAME
+    if headphones.CONFIG.SAB_PASSWORD:
+        params['ma_password'] = headphones.CONFIG.SAB_PASSWORD
+    if headphones.CONFIG.SAB_APIKEY:
+        params['apikey'] = headphones.CONFIG.SAB_APIKEY
+
+    if request_type == 'send_nzb' and headphones.CONFIG.SAB_CATEGORY:
+        params['cat'] = headphones.CONFIG.SAB_CATEGORY
+
+    params['output'] = 'json'
+
+    response = request.request_json(url, params=params, **kwargs)
+
+    if not response:
+        logger.error("Error connecting to SABnzbd on url: %s" % headphones.CONFIG.SAB_HOST)
+        return False
+    else:
+        logger.debug("Successfully connected to SABnzbd on url: %s" % headphones.CONFIG.SAB_HOST)
+        return response
+
 
 def sendNZB(nzb):
-
     params = {}
-
-    if headphones.SAB_USERNAME:
-        params['ma_username'] = headphones.SAB_USERNAME
-    if headphones.SAB_PASSWORD:
-        params['ma_password'] = headphones.SAB_PASSWORD
-    if headphones.SAB_APIKEY:
-        params['apikey'] = headphones.SAB_APIKEY
-    if headphones.SAB_CATEGORY:
-        params['cat'] = headphones.SAB_CATEGORY
-
     # if it's a normal result we just pass SAB the URL
     if nzb.resultType == "nzb":
         # for newzbin results send the ID to sab specifically
         if nzb.provider.getID() == 'newzbin':
             id = nzb.provider.getIDFromURL(nzb.url)
             if not id:
-                logger.info("Unable to send NZB to sab, can't find ID in URL "+str(nzb.url))
+                logger.info("Unable to send NZB to sab, can't find ID in URL " + str(nzb.url))
                 return False
             params['mode'] = 'addid'
             params['name'] = id
@@ -62,101 +77,42 @@ def sendNZB(nzb):
         # Sanitize the file a bit, since we can only use ascii chars with MultiPartPostHandler
         nzbdata = helpers.latinToAscii(nzb.extraInfo[0])
         params['mode'] = 'addfile'
-        multiPartParams = {"nzbfile": (helpers.latinToAscii(nzb.name)+".nzb", nzbdata)}
+        files = {"nzbfile": (helpers.latinToAscii(nzb.name) + ".nzb", nzbdata)}
+        headers = {'User-Agent': USER_AGENT}
 
-    if not headphones.SAB_HOST.startswith('http'):
-        headphones.SAB_HOST = 'http://' + headphones.SAB_HOST
+    logger.info("Attempting to connect to SABnzbd on url: %s" % headphones.CONFIG.SAB_HOST)
+    if nzb.resultType == "nzb":
+        response = sab_api_call('send_nzb', params=params)
+    elif nzb.resultType == "nzbdata":
+        cookies = cookielib.CookieJar()
+        response = sab_api_call('send_nzb', params=params, method="post", files=files,
+                                cookies=cookies, headers=headers)
 
-    if headphones.SAB_HOST.endswith('/'):
-        headphones.SAB_HOST = headphones.SAB_HOST[0:len(headphones.SAB_HOST)-1]
-
-    url = headphones.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
-
-    try:
-
-        if nzb.resultType == "nzb":
-            f = urllib.urlopen(url)
-        elif nzb.resultType == "nzbdata":
-            cookies = cookielib.CookieJar()
-            opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies),
-                                          MultipartPostHandler.MultipartPostHandler)
-
-            req = urllib2.Request(url,
-                                  multiPartParams,
-                                  headers={'User-Agent': USER_AGENT})
-
-            f = opener.open(req)
-
-    except (EOFError, IOError), e:
-        logger.error(u"Unable to connect to SAB with URL: %s" % url)
-        return False
-
-    except httplib.InvalidURL, e:
-        logger.error(u"Invalid SAB host, check your config. Current host: %s" % headphones.SAB_HOST)
-        return False
-        
-    except Exception, e:
-        logger.error(u"Error: " + str(e))
-        return False
-        
-    if f == None:
+    if not response:
         logger.info(u"No data returned from SABnzbd, NZB not sent")
         return False
 
-    try:
-        result = f.readlines()
-    except Exception, e:
-        logger.info(u"Error trying to get result from SAB, NZB not sent: ")
-        return False
-
-    if len(result) == 0:
-        logger.info(u"No data returned from SABnzbd, NZB not sent")
-        return False
-
-    sabText = result[0].strip()
-
-    logger.info(u"Result text from SAB: " + sabText)
-
-    if sabText == "ok":
-        logger.info(u"NZB sent to SAB successfully")
+    if response['status']:
+        logger.info(u"NZB sent to SABnzbd successfully")
         return True
-    elif sabText == "Missing authentication":
-        logger.info(u"Incorrect username/password sent to SAB, NZB not sent")
-        return False
     else:
-        logger.info(u"Unknown failure sending NZB to sab. Return text is: " + sabText)
+        logger.error(u"Error sending NZB to SABnzbd: %s" % response['error'])
         return False
-        
+
+
 def checkConfig():
+    params = {'mode': 'get_config',
+              'section': 'misc',
+              }
 
-    params = { 'mode' : 'get_config', 
-               'section' : 'misc' 
-               }
+    config_options = sab_api_call(params=params)
 
-    if headphones.SAB_USERNAME:
-        params['ma_username'] = headphones.SAB_USERNAME
-    if headphones.SAB_PASSWORD:
-        params['ma_password'] = headphones.SAB_PASSWORD
-    if headphones.SAB_APIKEY:
-        params['apikey'] = headphones.SAB_APIKEY
-
-    if not headphones.SAB_HOST.startswith('http'):
-        headphones.SAB_HOST = 'http://' + headphones.SAB_HOST
-
-    if headphones.SAB_HOST.endswith('/'):
-        headphones.SAB_HOST = headphones.SAB_HOST[0:len(headphones.SAB_HOST)-1]
-    
-    url = headphones.SAB_HOST + "/" + "api?" + urllib.urlencode(params)
-    
-    try:
-        f = urllib.urlopen(url).read()
-    except Exception, e:
-        logger.warn("Unable to read SABnzbd config file - cannot determine renaming options (might affect auto & forced post processing)")
+    if not config_options:
+        logger.warn(
+            "Unable to read SABnzbd config file - cannot determine renaming options (might affect auto & forced post processing)")
         return (0, 0)
-        
-    config_options = ast.literal_eval(f)
-    
-    replace_spaces = config_options['misc']['replace_spaces']
-    replace_dots = config_options['misc']['replace_dots']
-    
+
+    replace_spaces = config_options['config']['misc']['replace_spaces']
+    replace_dots = config_options['config']['misc']['replace_dots']
+
     return (replace_spaces, replace_dots)

headphones/searcher_rutracker.py

@@ -1,349 +0,0 @@
-#!/usr/bin/env python
-# coding=utf-8
-
-# Headphones rutracker.org search
-# Functions called from searcher.py
-
-from headphones import logger, db, utorrent
-
-from bencode import bencode as bencode, bdecode
-from urlparse import urlparse
-from bs4 import BeautifulSoup
-from tempfile import mkdtemp
-from hashlib import sha1
-
-import headphones
-import requests
-import cookielib
-import urllib2
-import urllib
-import re
-import os
-
-class Rutracker():
-
-    logged_in = False
-
-    # Stores a number of login attempts to prevent recursion.
-    #login_counter = 0
-
-    def __init__(self):
-
-        self.cookiejar = cookielib.CookieJar()
-        self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(self.cookiejar))
-        urllib2.install_opener(self.opener)
-
-    def login(self, login, password):
-        """Implements tracker login procedure."""
-
-        self.logged_in = False
-
-        if login is None or password is None:
-            return False
-
-        #self.login_counter += 1
-
-        # No recursion wanted.
-        #if self.login_counter > 1:
-        #    return False
-
-        params = urllib.urlencode({"login_username" : login,
-                                   "login_password" : password,
-                                   "login" : "Вход"})
-
-        try:
-            self.opener.open("http://login.rutracker.org/forum/login.php", params)
-        except Exception:
-            pass
-
-        # Check if we're logged in
-        for cookie in self.cookiejar:
-            if cookie.name == 'bb_data':
-                self.logged_in = True
-
-        return self.logged_in
-
-    def searchurl(self, artist, album, year, format):
-        """
-        Return the search url
-        """
-
-        # Build search url
-        searchterm = ''
-        if artist != 'Various Artists':
-            searchterm = artist
-            searchterm = searchterm + ' '
-        searchterm = searchterm + album
-        searchterm = searchterm + ' '
-        searchterm = searchterm + year
-
-        providerurl = "http://rutracker.org/forum/tracker.php"
-
-        if format == 'lossless':
-            format = '+lossless'
-        elif format == 'lossless+mp3':
-            format = '+lossless||mp3||aac'
-        else:
-            format = '+mp3||aac'
-
-        # sort by size, descending.
-        sort = '&o=7&s=2'
-
-        searchurl = "%s?nm=%s%s%s" % (providerurl, urllib.quote(searchterm), format, sort)
-
-        return searchurl
-
-    def search(self, searchurl, maxsize, minseeders, albumid):
-        """
-        Parse the search results and return valid torrent list
-        """
-
-        titles = []
-        urls = []
-        seeders = []
-        sizes = []
-        torrentlist = []
-        rulist = []
-
-        try:
-
-            page = self.opener.open(searchurl, timeout=60)
-            soup = BeautifulSoup(page.read())
-
-            # Debug
-            #logger.debug (soup.prettify())
-
-            # Title
-            for link in soup.find_all('a', attrs={'class' : 'med tLink hl-tags bold'}):
-                title = link.get_text()
-                titles.append(title)
-
-            # Download URL
-            for link in soup.find_all('a', attrs={'class' : 'small tr-dl dl-stub'}):
-                url = link.get('href')
-                urls.append(url)
-
-            # Seeders
-            for link in soup.find_all('b', attrs={'class' : 'seedmed'}):
-                seeder = link.get_text()
-                seeders.append(seeder)
-
-            # Size
-            for link in soup.find_all('td', attrs={'class' : 'row4 small nowrap tor-size'}):
-                size = link.u.string
-                sizes.append(size)
-
-        except :
-            pass
-
-        # Combine lists
-        torrentlist = zip(titles, urls, seeders, sizes)
-
-        # return if nothing found
-        if not torrentlist:
-            return False
-
-        # don't bother checking track counts anymore, let searcher filter instead
-        # leave code in just in case
-        check_track_count = False
-
-        if check_track_count:
-
-            # get headphones track count for album, return if not found
-            myDB = db.DBConnection()
-            tracks = myDB.select('SELECT * from tracks WHERE AlbumID=?', [albumid])
-            hptrackcount = len(tracks)
-
-            if not hptrackcount:
-                logger.info('headphones track info not found, cannot compare to torrent')
-                return False
-
-            # Return all valid entries, ignored, required words now checked in searcher.py
-
-            #unwantedlist = ['promo', 'vinyl', '[lp]', 'songbook', 'tvrip', 'hdtv', 'dvd']
-
-            formatlist = ['ape', 'flac', 'ogg', 'm4a', 'aac', 'mp3', 'wav', 'aif']
-            deluxelist = ['deluxe', 'edition', 'japanese', 'exclusive']
-
-        for torrent in torrentlist:
-
-            returntitle = torrent[0].encode('utf-8')
-            url = torrent[1]
-            seeders = torrent[2]
-            size = torrent[3]
-
-            if int(size) <= maxsize and int(seeders) >= minseeders:
-
-                #Torrent topic page
-                torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
-                topicurl = 'http://rutracker.org/forum/viewtopic.php?t=' + torrent_id
-
-                # add to list
-                if not check_track_count:
-                    valid = True
-                else:
-
-                    # Check torrent info
-                    self.cookiejar.set_cookie(cookielib.Cookie(version=0, name='bb_dl', value=torrent_id, port=None, port_specified=False, domain='.rutracker.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))
-
-                    # Debug
-                    #for cookie in self.cookiejar:
-                    #    logger.debug ('Cookie: %s' % cookie)
-
-                    try:
-                        page = self.opener.open(url)
-                        torrent = page.read()
-                        if torrent:
-                            decoded = bdecode(torrent)
-                            metainfo = decoded['info']
-                        page.close ()
-                    except Exception, e:
-                        logger.error('Error getting torrent: %s' % e)
-                        return False
-
-                    # get torrent track count and check for cue
-                    trackcount = 0
-                    cuecount = 0
-
-                    if 'files' in metainfo: # multi
-                        for pathfile in metainfo['files']:
-                            path = pathfile['path']
-                            for file in path:
-                                if any(file.lower().endswith('.' + x.lower()) for x in formatlist):
-                                    trackcount += 1
-                                if '.cue' in file:
-                                    cuecount += 1
-
-                    title = returntitle.lower()
-                    logger.debug ('torrent title: %s' % title)
-                    logger.debug ('headphones trackcount: %s' % hptrackcount)
-                    logger.debug ('rutracker trackcount: %s' % trackcount)
-
-                    # If torrent track count less than headphones track count, and there's a cue, then attempt to get track count from log(s)
-                    # This is for the case where we have a single .flac/.wav which can be split by cue
-                    # Not great, but shouldn't be doing this too often
-                    totallogcount = 0
-                    if trackcount < hptrackcount and cuecount > 0 and cuecount < hptrackcount:
-                        page = self.opener.open(topicurl, timeout=60)
-                        soup = BeautifulSoup(page.read())
-                        findtoc = soup.find_all(text='TOC of the extracted CD')
-                        if not findtoc:
-                            findtoc = soup.find_all(text='TOC извлечённого CD')
-                        for toc in findtoc:
-                            logcount = 0
-                            for toccontent in toc.find_all_next(text=True):
-                                cut_string = toccontent.split('|')
-                                new_string = cut_string[0].lstrip().rstrip()
-                                if new_string == '1' or new_string == '01':
-                                    logcount = 1
-                                elif logcount > 0:
-                                    if new_string.isdigit():
-                                        logcount += 1
-                                    else:
-                                        break
-                            totallogcount = totallogcount + logcount
-
-                    if totallogcount > 0:
-                        trackcount = totallogcount
-                        logger.debug ('rutracker logtrackcount: %s' % totallogcount)
-
-                    # If torrent track count = hp track count then return torrent,
-                    # if greater, check for deluxe/special/foreign editions
-                    # if less, then allow if it's a single track with a cue
-                    valid = False
-
-                    if trackcount == hptrackcount:
-                        valid = True
-                    elif trackcount > hptrackcount:
-                        if any(deluxe in title for deluxe in deluxelist):
-                            valid = True
-
-                # Add to list
-                if valid:
-                    rulist.append((returntitle, size, topicurl))
-                else:
-                    if topicurl:
-                        logger.info(u'<a href="%s">Torrent</a> found with %s tracks but the selected headphones release has %s tracks, skipping for rutracker.org' % (topicurl, trackcount, hptrackcount))
-            else:
-                logger.info('%s is larger than the maxsize or has too little seeders for this category, skipping. (Size: %i bytes, Seeders: %i)' % (returntitle, int(size), int(seeders)))
-
-        return rulist
-
-    def get_torrent(self, url, savelocation=None):
-
-        torrent_id = dict([part.split('=') for part in urlparse(url)[4].split('&')])['t']
-        self.cookiejar.set_cookie(cookielib.Cookie(version=0, name='bb_dl', value=torrent_id, port=None, port_specified=False, domain='.rutracker.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))
-        downloadurl = 'http://dl.rutracker.org/forum/dl.php?t=' + torrent_id
-        torrent_name = torrent_id + '.torrent'
-
-        try:
-            prev = os.umask(headphones.UMASK)
-            page = self.opener.open(downloadurl)
-            torrent = page.read()
-            decoded = bdecode(torrent)
-            metainfo = decoded['info']
-            tor_hash = sha1(bencode(metainfo)).hexdigest()
-            if savelocation:
-                download_path = os.path.join(savelocation, torrent_name)
-            else:
-                tempdir = mkdtemp(suffix='_rutracker_torrents')
-                download_path = os.path.join(tempdir, torrent_name)
-
-            with open(download_path, 'wb') as f:
-                f.write(torrent)
-            os.umask(prev)
-
-            # Add file to utorrent
-            if headphones.TORRENT_DOWNLOADER == 2:
-                self.utorrent_add_file(download_path)
-
-        except Exception as e:
-            logger.error('Error getting torrent: %s', e)
-            return False
-
-        return download_path, tor_hash
-
-    #TODO get this working in utorrent.py
-    def utorrent_add_file(self, filename):
-
-        host = headphones.UTORRENT_HOST
-        if not host.startswith('http'):
-            host = 'http://' + host
-        if host.endswith('/'):
-            host = host[:-1]
-        if host.endswith('/gui'):
-            host = host[:-4]
-
-        base_url = host
-        username = headphones.UTORRENT_USERNAME
-        password = headphones.UTORRENT_PASSWORD
-
-        session = requests.Session()
-        url = base_url + '/gui/'
-        session.auth = (username, password)
-
-        try:
-            r = session.get(url + 'token.html')
-        except Exception:
-            logger.exception('Error getting token')
-            return
-
-        if r.status_code == '401':
-            logger.debug('Error reaching utorrent')
-            return
-
-        regex = re.search(r'.+>([^<]+)</div></html>', r.text)
-        if regex is None:
-            logger.debug('Error reading token')
-            return
-
-        session.params = {'token': regex.group(1)}
-
-        with open(filename, 'rb') as f:
-            try:
-                session.post(url, params={'action': 'add-file'},
-                    files={'torrent_file': f})
-            except Exception:
-                logger.exception('Error adding file to utorrent')
-                return
-

headphones/torrentfinished.py

@@ -14,18 +14,21 @@
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
 import threading
-import headphones
+
 from headphones import db, utorrent, transmission, logger
+import headphones
 
 postprocessor_lock = threading.Lock()
 
-# Remove Torrent + data if Post Processed and finished Seeding
+
 def checkTorrentFinished():
+    """
+    Remove Torrent + data if Post Processed and finished Seeding
+    """
 
     logger.info("Checking if any torrents have finished seeding and can be removed")
 
     with postprocessor_lock:
-
         myDB = db.DBConnection()
         results = myDB.select('SELECT * from snatched WHERE Status="Seed_Processed"')
 
@@ -33,10 +36,13 @@ def checkTorrentFinished():
             hash = album['FolderName']
             albumid = album['AlbumID']
             torrent_removed = False
-            if headphones.TORRENT_DOWNLOADER == 1:
+            if headphones.CONFIG.TORRENT_DOWNLOADER == 1:
                 torrent_removed = transmission.removeTorrent(hash, True)
             else:
                 torrent_removed = utorrent.removeTorrent(hash, True)
 
             if torrent_removed:
-                myDB.action('DELETE from snatched WHERE status = "Seed_Processed" and AlbumID=?', [albumid])
+                myDB.action('DELETE from snatched WHERE status = "Seed_Processed" and AlbumID=?',
+                            [albumid])
+
+    logger.info("Checking finished torrents completed")

headphones/transmission.py

@@ -13,44 +13,46 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-from headphones import logger, notifiers, request
-
-import re
 import time
 import json
 import base64
+import urlparse
+
+from headphones import logger, request
 import headphones
 
+
 # This is just a simple script to send torrents to transmission. The
 # intention is to turn this into a class where we can check the state
 # of the download, set the download dir, etc.
 # TODO: Store the session id so we don't need to make 2 calls
 #       Store torrent id so we can check up on it
 
-def addTorrent(link):
+
+def addTorrent(link, data=None):
     method = 'torrent-add'
 
-    if link.endswith('.torrent'):
-        with open(link, 'rb') as f:
-            metainfo = str(base64.b64encode(f.read()))
-        arguments = {'metainfo': metainfo, 'download-dir':headphones.DOWNLOAD_TORRENT_DIR}
+    if link.endswith('.torrent') and not link.startswith('http') or data:
+        if data:
+            metainfo = str(base64.b64encode(data))
+        else:
+            with open(link, 'rb') as f:
+                metainfo = str(base64.b64encode(f.read()))
+        arguments = {'metainfo': metainfo, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}
     else:
-        arguments = {'filename': link, 'download-dir': headphones.DOWNLOAD_TORRENT_DIR}
+        arguments = {'filename': link, 'download-dir': headphones.CONFIG.DOWNLOAD_TORRENT_DIR}
 
-    response = torrentAction(method,arguments)
+    response = torrentAction(method, arguments)
 
     if not response:
         return False
 
     if response['result'] == 'success':
         if 'torrent-added' in response['arguments']:
-            name = response['arguments']['torrent-added']['name']
             retid = response['arguments']['torrent-added']['hashString']
         elif 'torrent-duplicate' in response['arguments']:
-            name = response['arguments']['torrent-duplicate']['name']
             retid = response['arguments']['torrent-duplicate']['hashString']
         else:
-            name = link
             retid = False
 
         logger.info(u"Torrent sent to Transmission successfully")
@@ -60,9 +62,10 @@ def addTorrent(link):
         logger.info('Transmission returned status %s' % response['result'])
         return False
 
+
 def getTorrentFolder(torrentid):
     method = 'torrent-get'
-    arguments = { 'ids': torrentid, 'fields': ['name','percentDone']}
+    arguments = {'ids': torrentid, 'fields': ['name', 'percentDone']}
 
     response = torrentAction(method, arguments)
     percentdone = response['arguments']['torrents'][0]['percentDone']
@@ -70,8 +73,8 @@ def getTorrentFolder(torrentid):
 
     tries = 1
 
-    while percentdone == 0  and tries <10:
-        tries+=1
+    while percentdone == 0 and tries < 10:
+        tries += 1
         time.sleep(5)
         response = torrentAction(method, arguments)
         percentdone = response['arguments']['torrents'][0]['percentDone']
@@ -80,6 +83,7 @@ def getTorrentFolder(torrentid):
 
     return torrent_folder_name
 
+
 def setSeedRatio(torrentid, ratio):
     method = 'torrent-set'
     if ratio != 0:
@@ -91,10 +95,10 @@ def setSeedRatio(torrentid, ratio):
     if not response:
         return False
 
-def removeTorrent(torrentid, remove_data = False):
 
+def removeTorrent(torrentid, remove_data=False):
     method = 'torrent-get'
-    arguments = { 'ids': torrentid, 'fields': ['isFinished', 'name']}
+    arguments = {'ids': torrentid, 'fields': ['isFinished', 'name']}
 
     response = torrentAction(method, arguments)
     if not response:
@@ -114,18 +118,18 @@ def removeTorrent(torrentid, remove_data = False):
             response = torrentAction(method, arguments)
             return True
         else:
-            logger.info('%s has not finished seeding yet, torrent will not be removed, will try again on next run' % name)
+            logger.info(
+                '%s has not finished seeding yet, torrent will not be removed, will try again on next run' % name)
     except:
         return False
 
     return False
 
-def torrentAction(method, arguments):
 
-    host = headphones.TRANSMISSION_HOST
-    username = headphones.TRANSMISSION_USERNAME
-    password = headphones.TRANSMISSION_PASSWORD
-    sessionid = None
+def torrentAction(method, arguments):
+    host = headphones.CONFIG.TRANSMISSION_HOST
+    username = headphones.CONFIG.TRANSMISSION_USERNAME
+    password = headphones.CONFIG.TRANSMISSION_PASSWORD
 
     if not host.startswith('http'):
         host = 'http://' + host
@@ -133,28 +137,22 @@ def torrentAction(method, arguments):
     if host.endswith('/'):
         host = host[:-1]
 
-    # Either the host ends with a port, or some directory, or rpc
-    # If it ends with /rpc we don't have to do anything
-    # If it ends with a port we add /transmission/rpc
-    # anything else we just add rpc
-    if not host.endswith('/rpc'):
-        # Check if it ends in a port number
-        i = host.rfind(':')
-        if i >= 0:
-            possible_port = host[i+1:]
-            try:
-                port = int(possible_port)
-                host = host + "/transmission/rpc"
-            except ValueError:
-                host = host + "/rpc"
-        else:
-            logger.error('Transmission port missing')
-            return
+    # Fix the URL. We assume that the user does not point to the RPC endpoint,
+    # so add it if it is missing.
+    parts = list(urlparse.urlparse(host))
+
+    if not parts[0] in ("http", "https"):
+        parts[0] = "http"
+
+    if not parts[2].endswith("/rpc"):
+        parts[2] += "/transmission/rpc"
+
+    host = urlparse.urlunparse(parts)
 
     # Retrieve session id
     auth = (username, password) if username and password else None
-
-    response = request.request_response(host, auth=auth, whitelist_status_code=[401, 409])
+    response = request.request_response(host, auth=auth,
+                                        whitelist_status_code=[401, 409])
 
     if response is None:
         logger.error("Error gettings Transmission session ID")
@@ -163,23 +161,27 @@ def torrentAction(method, arguments):
     # Parse response
     if response.status_code == 401:
         if auth:
-            logger.error("Username and/or password not accepted by Transmission")
+            logger.error("Username and/or password not accepted by " \
+                         "Transmission")
         else:
             logger.error("Transmission authorization required")
 
         return
     elif response.status_code == 409:
-        sessionid = response.headers['x-transmission-session-id']
+        session_id = response.headers['x-transmission-session-id']
 
-    if not sessionid:
-        logger.error("Error getting Session ID from Transmission")
-        return
+        if not session_id:
+            logger.error("Expected a Session ID from Transmission")
+            return
 
     # Prepare next request
-    headers = { 'x-transmission-session-id': sessionid }
-    data = { 'method': method, 'arguments': arguments }
+    headers = {'x-transmission-session-id': session_id}
+    data = {'method': method, 'arguments': arguments}
 
-    response = request.request_json(host, method="post", data=json.dumps(data), headers=headers, auth=auth)
+    response = request.request_json(host, method="POST", data=json.dumps(data),
+                                    headers=headers, auth=auth)
+
+    print response
 
     if not response:
         logger.error("Error sending torrent to Transmission")

headphones/updater.py

@@ -13,19 +13,17 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import headphones
-
 from headphones import logger, db, importer
 
-def dbUpdate(forcefull=False):
 
+def dbUpdate(forcefull=False):
     myDB = db.DBConnection()
 
-    activeartists = myDB.select('SELECT ArtistID, ArtistName from artists WHERE Status="Active" or Status="Loading" order by LastUpdated ASC')
-    logger.info('Starting update for %i active artists' % len(activeartists))
-
-    for artist in activeartists:
+    active_artists = myDB.select(
+        'SELECT ArtistID, ArtistName from artists WHERE Status="Active" or Status="Loading" order by LastUpdated ASC')
+    logger.info('Starting update for %i active artists', len(active_artists))
 
+    for artist in active_artists:
         artistid = artist[0]
         importer.addArtisttoDB(artistid=artistid, extrasonly=False, forcefull=forcefull)
 

headphones/utorrent.py

@@ -13,22 +13,27 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import urllib, urllib2, urlparse, cookielib
-import json, re, os, time
-
-import headphones
-
-from headphones import logger
+import urllib
+import json
+import time
 from collections import namedtuple
+import urllib2
+import urlparse
+import cookielib
+
+import re
+import os
+import headphones
+from headphones import logger
+
 
 class utorrentclient(object):
-
     TOKEN_REGEX = "<div id='token' style='display:none;'>([^<>]+)</div>"
     UTSetting = namedtuple("UTSetting", ["name", "int", "str", "access"])
 
-    def __init__(self, base_url = None, username = None, password = None,):
+    def __init__(self, base_url=None, username=None, password=None, ):
 
-        host = headphones.UTORRENT_HOST
+        host = headphones.CONFIG.UTORRENT_HOST
         if not host.startswith('http'):
             host = 'http://' + host
 
@@ -39,16 +44,16 @@ class utorrentclient(object):
             host = host[:-4]
 
         self.base_url = host
-        self.username = headphones.UTORRENT_USERNAME
-        self.password = headphones.UTORRENT_PASSWORD
+        self.username = headphones.CONFIG.UTORRENT_USERNAME
+        self.password = headphones.CONFIG.UTORRENT_PASSWORD
         self.opener = self._make_opener('uTorrent', self.base_url, self.username, self.password)
         self.token = self._get_token()
-        #TODO refresh token, when necessary
+        # TODO refresh token, when necessary
 
     def _make_opener(self, realm, base_url, username, password):
         """uTorrent API need HTTP Basic Auth and cookie support for token verify."""
         auth = urllib2.HTTPBasicAuthHandler()
-        auth.add_password(realm=realm,uri=base_url,user=username,passwd=password)
+        auth.add_password(realm=realm, uri=base_url, user=username, passwd=password)
         opener = urllib2.build_opener(auth)
         urllib2.install_opener(opener)
 
@@ -66,6 +71,7 @@ class utorrentclient(object):
         except urllib2.HTTPError as err:
             logger.debug('URL: ' + str(url))
             logger.debug('Error getting Token. uTorrent responded with error: ' + str(err))
+            return
         match = re.search(utorrentclient.TOKEN_REGEX, response.read())
         return match.group(1)
 
@@ -75,7 +81,7 @@ class utorrentclient(object):
         return self._action(params)
 
     def add_url(self, url):
-        #can receive magnet or normal .torrent link
+        # can receive magnet or normal .torrent link
         params = [('action', 'add-url'), ('s', url)]
         return self._action(params)
 
@@ -132,7 +138,7 @@ class utorrentclient(object):
             return settings[key]
         return settings
 
-    def remove(self, hash, remove_data = False):
+    def remove(self, hash, remove_data=False):
         if remove_data:
             params = [('action', 'removedata'), ('hash', hash)]
         else:
@@ -140,6 +146,10 @@ class utorrentclient(object):
         return self._action(params)
 
     def _action(self, params, body=None, content_type=None):
+
+        if not self.token:
+            return
+
         url = self.base_url + '/gui/' + '?token=' + self.token + '&' + urllib.urlencode(params)
         request = urllib2.Request(url)
 
@@ -156,38 +166,43 @@ class utorrentclient(object):
             logger.debug('URL: ' + str(url))
             logger.debug('uTorrent webUI raised the following error: ' + str(err))
 
+
 def labelTorrent(hash):
-    label = headphones.UTORRENT_LABEL
+    label = headphones.CONFIG.UTORRENT_LABEL
     uTorrentClient = utorrentclient()
     if label:
-        uTorrentClient.setprops(hash,'label',label)
+        uTorrentClient.setprops(hash, 'label', label)
 
-def removeTorrent(hash, remove_data = False):
+
+def removeTorrent(hash, remove_data=False):
     uTorrentClient = utorrentclient()
     status, torrentList = uTorrentClient.list()
     torrents = torrentList['torrents']
     for torrent in torrents:
-        if torrent[0].lower() == hash:
+        if torrent[0].upper() == hash.upper():
             if torrent[21] == 'Finished':
                 logger.info('%s has finished seeding, removing torrent and data' % torrent[2])
                 uTorrentClient.remove(hash, remove_data)
                 return True
             else:
-                logger.info('%s has not finished seeding yet, torrent will not be removed, will try again on next run' % torrent[2])
+                logger.info(
+                    '%s has not finished seeding yet, torrent will not be removed, will try again on next run' %
+                    torrent[2])
                 return False
     return False
 
+
 def setSeedRatio(hash, ratio):
     uTorrentClient = utorrentclient()
     uTorrentClient.setprops(hash, 'seed_override', '1')
     if ratio != 0:
-        uTorrentClient.setprops(hash,'seed_ratio', ratio * 10)
+        uTorrentClient.setprops(hash, 'seed_ratio', ratio * 10)
     else:
         # TODO passing -1 should be unlimited
-        uTorrentClient.setprops(hash,'seed_ratio', -10)
+        uTorrentClient.setprops(hash, 'seed_ratio', -10)
+
 
 def dirTorrent(hash, cacheid=None, return_name=None):
-
     uTorrentClient = utorrentclient()
 
     if not cacheid:
@@ -204,7 +219,7 @@ def dirTorrent(hash, cacheid=None, return_name=None):
     cacheid = torrentList['torrentc']
 
     for torrent in torrents:
-        if (torrent[0].lower() == hash):
+        if torrent[0].upper() == hash.upper():
             if not return_name:
                 return torrent[26], cacheid
             else:
@@ -212,37 +227,42 @@ def dirTorrent(hash, cacheid=None, return_name=None):
 
     return None, None
 
-def addTorrent(link, hash):
+
+def addTorrent(link):
     uTorrentClient = utorrentclient()
+    uTorrentClient.add_url(link)
+
+
+def getFolder(hash):
 
     # Get Active Directory from settings
     active_dir, completed_dir = getSettingsDirectories()
 
     if not active_dir:
-        logger.error('Could not get "Put new downloads in:" directory from uTorrent settings, please ensure it is set')
+        logger.error(
+            'Could not get "Put new downloads in:" directory from uTorrent settings, please ensure it is set')
         return None
 
-    uTorrentClient.add_url(link)
-
     # Get Torrent Folder Name
     torrent_folder, cacheid = dirTorrent(hash)
 
     # If there's no folder yet then it's probably a magnet, try until folder is populated
     if torrent_folder == active_dir or not torrent_folder:
         tries = 1
-        while (torrent_folder == active_dir or torrent_folder == None) and tries <= 10:
+        while (torrent_folder == active_dir or torrent_folder is None) and tries <= 10:
             tries += 1
             time.sleep(6)
             torrent_folder, cacheid = dirTorrent(hash, cacheid)
 
     if torrent_folder == active_dir or not torrent_folder:
         torrent_folder, cacheid = dirTorrent(hash, cacheid, return_name=True)
-        labelTorrent(hash)
         return torrent_folder
     else:
-        labelTorrent(hash)
+        if headphones.SYS_PLATFORM != "win32":
+            torrent_folder = torrent_folder.replace('\\', '/')
         return os.path.basename(os.path.normpath(torrent_folder))
 
+
 def getSettingsDirectories():
     uTorrentClient = utorrentclient()
     settings = uTorrentClient.get_settings()
@@ -253,4 +273,3 @@ def getSettingsDirectories():
     if 'dir_completed_download' in settings:
         completed = settings['dir_completed_download'][2]
     return active, completed
-

headphones/version.py

@@ -1 +1 @@
-HEADPHONES_VERSION = "master"
+HEADPHONES_VERSION = "master"

headphones/versioncheck.py

@@ -13,19 +13,19 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import re
-import os
 import tarfile
 import platform
-import headphones
 import subprocess
 
+import re
+import os
+import headphones
 from headphones import logger, version, request
 
-def runGit(args):
 
-    if headphones.GIT_PATH:
-        git_locations = ['"'+headphones.GIT_PATH+'"']
+def runGit(args):
+    if headphones.CONFIG.GIT_PATH:
+        git_locations = ['"' + headphones.CONFIG.GIT_PATH + '"']
     else:
         git_locations = ['git']
 
@@ -35,11 +35,12 @@ def runGit(args):
     output = err = None
 
     for cur_git in git_locations:
-        cmd = cur_git+' '+args
+        cmd = cur_git + ' ' + args
 
         try:
             logger.debug('Trying to execute: "' + cmd + '" with shell in ' + headphones.PROG_DIR)
-            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True, cwd=headphones.PROG_DIR)
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True,
+                                 cwd=headphones.PROG_DIR)
             output, err = p.communicate()
             output = output.strip()
 
@@ -59,8 +60,8 @@ def runGit(args):
 
     return (output, err)
 
-def getVersion():
 
+def getVersion():
     if version.HEADPHONES_VERSION.startswith('win32build'):
         headphones.INSTALL_TYPE = 'win'
 
@@ -82,16 +83,17 @@ def getVersion():
             logger.error('Output doesn\'t look like a hash, not using it')
             cur_commit_hash = None
 
-        if headphones.DO_NOT_OVERRIDE_GIT_BRANCH and headphones.GIT_BRANCH:
-            branch_name = headphones.GIT_BRANCH
+        if headphones.CONFIG.DO_NOT_OVERRIDE_GIT_BRANCH and headphones.CONFIG.GIT_BRANCH:
+            branch_name = headphones.CONFIG.GIT_BRANCH
 
         else:
             branch_name, err = runGit('rev-parse --abbrev-ref HEAD')
             branch_name = branch_name
 
-            if not branch_name and headphones.GIT_BRANCH:
-                logger.error('Could not retrieve branch name from git. Falling back to %s' % headphones.GIT_BRANCH)
-                branch_name = headphones.GIT_BRANCH
+            if not branch_name and headphones.CONFIG.GIT_BRANCH:
+                logger.error(
+                    'Could not retrieve branch name from git. Falling back to %s' % headphones.CONFIG.GIT_BRANCH)
+                branch_name = headphones.CONFIG.GIT_BRANCH
             if not branch_name:
                 logger.error('Could not retrieve branch name from git. Defaulting to master')
                 branch_name = 'master'
@@ -111,20 +113,23 @@ def getVersion():
             current_version = f.read().strip(' \n\r')
 
         if current_version:
-            return current_version, headphones.GIT_BRANCH
+            return current_version, headphones.CONFIG.GIT_BRANCH
         else:
             return None, 'master'
 
+
 def checkGithub():
     headphones.COMMITS_BEHIND = 0
 
     # Get the latest version available from github
     logger.info('Retrieving latest version information from GitHub')
-    url = 'https://api.github.com/repos/%s/headphones/commits/%s' % (headphones.GIT_USER, headphones.GIT_BRANCH)
+    url = 'https://api.github.com/repos/%s/headphones/commits/%s' % (
+    headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
     version = request.request_json(url, timeout=20, validator=lambda x: type(x) == dict)
 
     if version is None:
-        logger.warn('Could not get the latest version from GitHub. Are you running a local development version?')
+        logger.warn(
+            'Could not get the latest version from GitHub. Are you running a local development version?')
         return headphones.CURRENT_VERSION
 
     headphones.LATEST_VERSION = version['sha']
@@ -132,7 +137,8 @@ def checkGithub():
 
     # See how many commits behind we are
     if not headphones.CURRENT_VERSION:
-        logger.info('You are running an unknown version of Headphones. Run the updater to identify your version')
+        logger.info(
+            'You are running an unknown version of Headphones. Run the updater to identify your version')
         return headphones.LATEST_VERSION
 
     if headphones.LATEST_VERSION == headphones.CURRENT_VERSION:
@@ -140,8 +146,10 @@ def checkGithub():
         return headphones.LATEST_VERSION
 
     logger.info('Comparing currently installed version with latest GitHub version')
-    url = 'https://api.github.com/repos/%s/headphones/compare/%s...%s' % (headphones.GIT_USER, headphones.LATEST_VERSION, headphones.CURRENT_VERSION)
-    commits = request.request_json(url, timeout=20, whitelist_status_code=404, validator=lambda x: type(x) == dict)
+    url = 'https://api.github.com/repos/%s/headphones/compare/%s...%s' % (
+    headphones.CONFIG.GIT_USER, headphones.LATEST_VERSION, headphones.CURRENT_VERSION)
+    commits = request.request_json(url, timeout=20, whitelist_status_code=404,
+                                   validator=lambda x: type(x) == dict)
 
     if commits is None:
         logger.warn('Could not get commits behind from GitHub.')
@@ -155,18 +163,20 @@ def checkGithub():
         headphones.COMMITS_BEHIND = 0
 
     if headphones.COMMITS_BEHIND > 0:
-        logger.info('New version is available. You are %s commits behind' % headphones.COMMITS_BEHIND)
+        logger.info(
+            'New version is available. You are %s commits behind' % headphones.COMMITS_BEHIND)
     elif headphones.COMMITS_BEHIND == 0:
         logger.info('Headphones is up to date')
 
     return headphones.LATEST_VERSION
 
+
 def update():
     if headphones.INSTALL_TYPE == 'win':
         logger.info('Windows .exe updating not supported yet.')
 
     elif headphones.INSTALL_TYPE == 'git':
-        output, err = runGit('pull origin ' + headphones.GIT_BRANCH)
+        output, err = runGit('pull origin ' + headphones.CONFIG.GIT_BRANCH)
 
         if not output:
             logger.error('Couldn\'t download latest version')
@@ -177,22 +187,23 @@ def update():
                 logger.info('No update available, not updating')
                 logger.info('Output: ' + str(output))
             elif line.endswith('Aborting.'):
-                logger.error('Unable to update from git: '+line)
+                logger.error('Unable to update from git: ' + line)
                 logger.info('Output: ' + str(output))
 
     else:
-        tar_download_url = 'https://github.com/%s/headphones/tarball/%s' % (headphones.GIT_USER, headphones.GIT_BRANCH)
+        tar_download_url = 'https://github.com/%s/headphones/tarball/%s' % (
+        headphones.CONFIG.GIT_USER, headphones.CONFIG.GIT_BRANCH)
         update_dir = os.path.join(headphones.PROG_DIR, 'update')
         version_path = os.path.join(headphones.PROG_DIR, 'version.txt')
 
-        logger.info('Downloading update from: '+ tar_download_url)
+        logger.info('Downloading update from: ' + tar_download_url)
         data = request.request_content(tar_download_url)
 
         if not data:
             logger.error("Unable to retrieve new version from '%s', can't update", tar_download_url)
             return
 
-        download_name = headphones.GIT_BRANCH + '-github'
+        download_name = headphones.CONFIG.GIT_BRANCH + '-github'
         tar_download_path = os.path.join(headphones.PROG_DIR, download_name)
 
         # Save tar to disk
@@ -210,15 +221,16 @@ def update():
         os.remove(tar_download_path)
 
         # Find update dir name
-        update_dir_contents = [x for x in os.listdir(update_dir) if os.path.isdir(os.path.join(update_dir, x))]
+        update_dir_contents = [x for x in os.listdir(update_dir) if
+                               os.path.isdir(os.path.join(update_dir, x))]
         if len(update_dir_contents) != 1:
-            logger.error("Invalid update data, update failed: "+str(update_dir_contents))
+            logger.error("Invalid update data, update failed: " + str(update_dir_contents))
             return
         content_dir = os.path.join(update_dir, update_dir_contents[0])
 
         # walk temp folder and move files to main folder
         for dirname, dirnames, filenames in os.walk(content_dir):
-            dirname = dirname[len(content_dir)+1:]
+            dirname = dirname[len(content_dir) + 1:]
             for curfile in filenames:
                 old_path = os.path.join(content_dir, dirname, curfile)
                 new_path = os.path.join(headphones.PROG_DIR, dirname, curfile)
@@ -232,6 +244,8 @@ def update():
             with open(version_path, 'w') as f:
                 f.write(str(headphones.LATEST_VERSION))
         except IOError as e:
-            logger.error("Unable to write current version to version.txt, " \
-                "update not complete: ", e)
+            logger.error(
+                "Unable to write current version to version.txt, update not complete: %s",
+                e
+            )
             return

headphones/webstart.py

@@ -13,35 +13,35 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
 import sys
+
+import os
 import cherrypy
 import headphones
-
 from headphones import logger
 from headphones.webserve import WebInterface
 from headphones.helpers import create_https_certificates
 
-def initialize(options=None):
-    if options is None:
-        options = {}
 
+def initialize(options):
     # HTTPS stuff stolen from sickbeard
     enable_https = options['enable_https']
     https_cert = options['https_cert']
     https_key = options['https_key']
 
     if enable_https:
-        # If either the HTTPS certificate or key do not exist, make some self-signed ones.
-        if not (https_cert and os.path.exists(https_cert)) or not (https_key and os.path.exists(https_key)):
+        # If either the HTTPS certificate or key do not exist, try to make
+        # self-signed ones.
+        if not (https_cert and os.path.exists(https_cert)) or not (
+            https_key and os.path.exists(https_key)):
             if not create_https_certificates(https_cert, https_key):
-                logger.warn(u"Unable to create cert/key files, disabling HTTPS")
-                headphones.ENABLE_HTTPS = False
+                logger.warn("Unable to create certificate and key. Disabling " \
+                            "HTTPS")
                 enable_https = False
 
         if not (os.path.exists(https_cert) and os.path.exists(https_key)):
-            logger.warn(u"Disabled HTTPS because of missing CERT and KEY files")
-            headphones.ENABLE_HTTPS = False
+            logger.warn("Disabled HTTPS because of missing certificate and " \
+                        "key.")
             enable_https = False
 
     options_dict = {
@@ -52,7 +52,7 @@ def initialize(options=None):
         'tools.encode.encoding': 'utf-8',
         'tools.decode.on': True,
         'log.screen': False,
-        'engine.autoreload_on': False,
+        'engine.autoreload.on': False,
     }
 
     if enable_https:
@@ -63,7 +63,7 @@ def initialize(options=None):
         protocol = "http"
 
     logger.info("Starting Headphones web server on %s://%s:%d/", protocol,
-        options['http_host'], options['http_port'])
+                options['http_host'], options['http_port'])
     cherrypy.config.update(options_dict)
 
     conf = {
@@ -71,56 +71,56 @@ def initialize(options=None):
             'tools.staticdir.root': os.path.join(headphones.PROG_DIR, 'data'),
             'tools.proxy.on': options['http_proxy']  # pay attention to X-Forwarded-Proto header
         },
-        '/interfaces':{
+        '/interfaces': {
             'tools.staticdir.on': True,
             'tools.staticdir.dir': "interfaces"
         },
-        '/images':{
+        '/images': {
             'tools.staticdir.on': True,
             'tools.staticdir.dir': "images"
         },
-        '/css':{
+        '/css': {
             'tools.staticdir.on': True,
             'tools.staticdir.dir': "css"
         },
-        '/js':{
+        '/js': {
             'tools.staticdir.on': True,
             'tools.staticdir.dir': "js"
         },
-        '/favicon.ico':{
+        '/favicon.ico': {
             'tools.staticfile.on': True,
             'tools.staticfile.filename': os.path.join(os.path.abspath(
                 os.curdir), "images" + os.sep + "favicon.ico")
         },
-        '/cache':{
+        '/cache': {
             'tools.staticdir.on': True,
-            'tools.staticdir.dir': headphones.CACHE_DIR
+            'tools.staticdir.dir': headphones.CONFIG.CACHE_DIR
         }
     }
 
     if options['http_password']:
-        logger.info("Web server authentication is enabled, username is '%s'", options['http_username'])
+        logger.info("Web server authentication is enabled, username is '%s'",
+                    options['http_username'])
 
         conf['/'].update({
             'tools.auth_basic.on': True,
             'tools.auth_basic.realm': 'Headphones web server',
-            'tools.auth_basic.checkpassword':  cherrypy.lib.auth_basic \
-                .checkpassword_dict({
-                    options['http_username']: options['http_password']
-                })
+            'tools.auth_basic.checkpassword': cherrypy.lib.auth_basic.checkpassword_dict({
+                options['http_username']: options['http_password']
+            })
         })
-        conf['/api'] = { 'tools.auth_basic.on': False }
-
+        conf['/api'] = {'tools.auth_basic.on': False}
 
     # Prevent time-outs
     cherrypy.engine.timeout_monitor.unsubscribe()
-    cherrypy.tree.mount(WebInterface(), options['http_root'], config=conf)
+    cherrypy.tree.mount(WebInterface(), str(options['http_root']), config=conf)
 
     try:
-        cherrypy.process.servers.check_port(options['http_host'], options['http_port'])
+        cherrypy.process.servers.check_port(str(options['http_host']), options['http_port'])
         cherrypy.server.start()
     except IOError:
-        sys.stderr.write('Failed to start on port: %i. Is something else running?\n' % (options['http_port']))
+        sys.stderr.write(
+            'Failed to start on port: %i. Is something else running?\n' % (options['http_port']))
         sys.exit(1)
 
-    cherrypy.server.wait()
+    cherrypy.server.wait()

init-scripts/init-alt.freebsd

@@ -27,17 +27,17 @@ rcvar=${name}_enable
 
 load_rc_config ${name}
 
-: ${headphones_enable:="NO"}
-: ${headphones_user:="_sabnzbd"}
-: ${headphones_dir:="/usr/local/headphones"}
-: ${headphones_chdir:="${headphones_dir}"}
-: ${headphones_pid:="${headphones_dir}/headphones.pid"}
-: ${headphones_conf:="${headphones_dir}/config.ini"}
+: "${headphones_enable:="NO"}"
+: "${headphones_user:="_sabnzbd"}"
+: "${headphones_dir:="/usr/local/headphones"}"
+: "${headphones_chdir:="${headphones_dir}"}"
+: "${headphones_pid:="${headphones_dir}/headphones.pid"}"
+: "${headphones_conf:="${headphones_dir}/config.ini"}"
 
 WGET="/usr/local/bin/wget" # You need wget for this script to safely shutdown Headphones.
 if [ -e "${headphones_conf}" ]; then
-        HOST=`grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_host"|perl -wple 's/^http_host = (.*)$/$1/'`
-        PORT=`grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_port"|perl -wple 's/^http_port = (.*)$/$1/'`
+        HOST=$(grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_host"|perl -wple 's/^http_host = (.*)$/$1/')
+        PORT=$(grep -A64 "\[General\]" "${headphones_conf}"|egrep "^http_port"|perl -wple 's/^http_port = (.*)$/$1/')
 fi
 
 status_cmd="${name}_status"
@@ -53,15 +53,15 @@ if [ ! -x "${WGET}" ]; then
 fi
 
 # Ensure user is root when running this script.
-if [ `id -u` != "0" ]; then
+if [ "$(id -u)" != "0" ]; then
   echo "Oops, you should be root before running this!"
   exit 1
 fi
 
 verify_headphones_pid() {
     # Make sure the pid corresponds to the Headphones process.
-    pid=`cat ${headphones_pid} 2>/dev/null`
-    ps -p ${pid} | grep -q "python ${headphones_dir}/Headphones.py"
+    pid=$(cat "${headphones_pid}" 2>/dev/null)
+    pgrep -F "${headphones_pid}" -q "python ${headphones_dir}/Headphones.py"
     return $?
 }
 
@@ -73,10 +73,10 @@ headphones_stop() {
     fi
     echo "Stopping $name"
     verify_headphones_pid
-    ${WGET} -O - -q --user=${SBUSR} --password=${SBPWD} "http://${HOST}:${PORT}/shutdown/" >/dev/null
+    ${WGET} -O - -q --user="${SBUSR}" --password="${SBPWD}" "http://${HOST}:${PORT}/shutdown/" >/dev/null
     
     if [ -n "${pid}" ]; then
-      wait_for_pids ${pid}
+      wait_for_pids "${pid}"
       echo "Stopped $name"
     fi
 }

init-scripts/init.freebsd

@@ -28,11 +28,11 @@ rcvar=${name}_enable
 
 load_rc_config ${name}
 
-: ${headphones_enable:="NO"}
-: ${headphones_user:="_sabnzbd"}
-: ${headphones_dir:="/usr/local/headphones"}
-: ${headphones_chdir:="${headphones_dir}"}
-: ${headphones_pid:="${headphones_dir}/headphones.pid"}
+: "${headphones_enable:="NO"}"
+: "${headphones_user:="_sabnzbd"}"
+: "${headphones_dir:="/usr/local/headphones"}"
+: "${headphones_chdir:="${headphones_dir}"}"
+: "${headphones_pid:="${headphones_dir}/headphones.pid"}"
 
 status_cmd="${name}_status"
 stop_cmd="${name}_stop"
@@ -41,15 +41,15 @@ command="/usr/sbin/daemon"
 command_args="-f -p ${headphones_pid} python ${headphones_dir}/Headphones.py ${headphones_flags} --quiet --nolaunch"
 
 # Ensure user is root when running this script.
-if [ `id -u` != "0" ]; then
+if [ "$(id -u)" != "0" ]; then
   echo "Oops, you should be root before running this!"
   exit 1
 fi
 
 verify_headphones_pid() {
     # Make sure the pid corresponds to the Headphones process.
-    pid=`cat ${headphones_pid} 2>/dev/null`
-    ps -p ${pid} | grep -q "python ${headphones_dir}/Headphones.py"
+    pid=$(cat "${headphones_pid}" 2>/dev/null)
+    pgrep -F "${headphones_pid}" -q "python ${headphones_dir}/Headphones.py"
     return $?
 }
 
@@ -58,7 +58,7 @@ headphones_stop() {
     echo "Stopping $name"
     verify_headphones_pid
     if [ -n "${pid}" ]; then
-      wait_for_pids ${pid}
+      wait_for_pids "${pid}"
       echo "Stopped"
     fi
 }

init-scripts/init.osx

@@ -3,16 +3,16 @@
 <plist version="1.0">
 <dict>
   <key>Label</key>
-  <string>com.headphones.headphones</string>
+  <string>headphones</string>
   <key>ProgramArguments</key>
   <array>
+       <!-- Modify these two lines if you need to to reflect your python location and Headphones install location -->
        <string>/usr/bin/python</string>
-       <string>/Applications/Headphones/headphones.py</string>
-       <string>--quiet</string>
-       <string>--daemon</string>
-       <string>--nolaunch</string>
+       <string>/Applications/Headphones/Headphones.py</string>
   </array>
   <key>RunAtLoad</key>
   <true/>
+  <key>KeepAlive</key>
+  <true/>
 </dict>
-</plist>
+</plist>

init-scripts/init.ubuntu

@@ -32,7 +32,6 @@
 ## HP_PIDFILE=      #$PID_FILE, the location of headphones.pid, the default is /var/run/headphones/headphones.pid
 ## PYTHON_BIN=      #$DAEMON, the location of the python binary, the default is /usr/bin/python
 ## HP_OPTS=         #$EXTRA_DAEMON_OPTS, extra cli option for headphones, i.e. " --config=/home/headphones/config.ini"
-## SSD_OPTS=        #$EXTRA_SSD_OPTS, extra start-stop-daemon option like " --group=users"
 ## HP_PORT=         #$PORT_OPTS, hardcoded port for the webserver, overrides value in config.ini
 ##
 ## EXAMPLE if want to run as different user
@@ -101,9 +100,6 @@ load_settings() {
         # Extra daemon option like: HP_OPTS=" --config=/home/headphones/config.ini"
         EXTRA_DAEMON_OPTS=${HP_OPTS-}
 
-        # Extra start-stop-daemon option like START_OPTS=" --group=users"
-        EXTRA_SSD_OPTS=${SSD_OPTS-}
-
         # Hardcoded port to run on, overrides config.ini settings
         [ -n "$HP_PORT" ] && {
             PORT_OPTS=" --port=${HP_PORT} "
@@ -114,7 +110,7 @@ load_settings() {
         SETTINGS_LOADED=TRUE
     fi
 
-    [ -x $DAEMON ] || {
+    [ -x "$DAEMON" ] || {
         log_warning_msg "$DESC: Can't execute daemon, aborting. See $DAEMON";
         return 1;}
 
@@ -125,8 +121,8 @@ load_settings || exit 0
 
 is_running () {
     # returns 1 when running, else 0.
-    if [ -e $PID_FILE ]; then
-      PID=`cat $PID_FILE`
+    if [ -e "$PID_FILE" ]; then
+      PID=$(cat "$PID_FILE")
 
       RET=$?
       [ $RET -gt 1 ] && exit 1 || return $RET
@@ -136,28 +132,28 @@ is_running () {
 }
 
 handle_pid () {
-    PID_PATH=`dirname $PID_FILE`
-    [ -d $PID_PATH ] || mkdir -p $PID_PATH && chown -R $RUN_AS $PID_PATH > /dev/null || {
+    PID_PATH=$(dirname "$PID_FILE")
+    [ -d "$PID_PATH" ] || mkdir -p "$PID_PATH" && chown -R "$RUN_AS" "$PID_PATH" > /dev/null || {
         log_warning_msg "$DESC: Could not create $PID_FILE, See $SETTINGS, aborting.";
         return 1;}
 
-    if [ -e $PID_FILE ]; then
-        PID=`cat $PID_FILE`
-        if ! kill -0 $PID > /dev/null 2>&1; then
+    if [ -e "$PID_FILE" ]; then
+        PID=$(cat "$PID_FILE")
+        if ! kill -0 "$PID" > /dev/null 2>&1; then
             log_warning_msg "Removing stale $PID_FILE"
-            rm $PID_FILE
+            rm "$PID_FILE"
         fi
     fi
 }
 
 handle_datadir () {
-    [ -d $DATA_DIR ] || mkdir -p $DATA_DIR && chown -R $RUN_AS $DATA_DIR > /dev/null || {
+    [ -d "$DATA_DIR" ] || mkdir -p "$DATA_DIR" && chown -R "$RUN_AS" "$DATA_DIR" > /dev/null || {
         log_warning_msg "$DESC: Could not create $DATA_DIR, See $SETTINGS, aborting.";
         return 1;}
 }
 
 handle_updates () {
-    chown -R $RUN_AS $APP_PATH > /dev/null || {
+    chown -R "$RUN_AS" "$APP_PATH" > /dev/null || {
         log_warning_msg "$DESC: $APP_PATH not writable by $RUN_AS for web-updates";
         return 0; }
 }
@@ -168,7 +164,7 @@ start_headphones () {
     handle_updates
     if ! is_running; then
         log_daemon_msg "Starting $DESC"
-        start-stop-daemon -o -d $APP_PATH -c $RUN_AS --start $EXTRA_SSD_OPTS --pidfile $PID_FILE --exec $DAEMON -- $DAEMON_OPTS
+        start-stop-daemon -o -d "$APP_PATH" -c "$RUN_AS" --start "$EXTRA_SSD"_OPTS --pidfile "$PID_FILE" --exec "$DAEMON" -- "$DAEMON_OPTS"
         check_retval
     else
         log_success_msg "$DESC: already running (pid $PID)"
@@ -178,7 +174,7 @@ start_headphones () {
 stop_headphones () {
     if is_running; then
         log_daemon_msg "Stopping $DESC"
-        start-stop-daemon -o --stop --pidfile $PID_FILE --retry 15
+        start-stop-daemon -o --stop --pidfile "$PID_FILE" --retry 15
         check_retval
     else
         log_success_msg "$DESC: not running"

lib/apscheduler/__init__.py

@@ -1,3 +1,5 @@
-version_info = (2, 0, 0, 'rc', 2)
-version = '.'.join(str(n) for n in version_info[:3])
-release = version + ''.join(str(n) for n in version_info[3:])
+version_info = (3, 0, 1)
+version = '3.0.1'
+release = '3.0.1'
+
+__version__ = release  # PEP 396

lib/apscheduler/events.py

@@ -1,63 +1,72 @@
-__all__ = ('EVENT_SCHEDULER_START', 'EVENT_SCHEDULER_SHUTDOWN',
-           'EVENT_JOBSTORE_ADDED', 'EVENT_JOBSTORE_REMOVED',
-           'EVENT_JOBSTORE_JOB_ADDED', 'EVENT_JOBSTORE_JOB_REMOVED',
-           'EVENT_JOB_EXECUTED', 'EVENT_JOB_ERROR', 'EVENT_JOB_MISSED',
-           'EVENT_ALL', 'SchedulerEvent', 'JobStoreEvent', 'JobEvent')
+__all__ = ('EVENT_SCHEDULER_START', 'EVENT_SCHEDULER_SHUTDOWN', 'EVENT_EXECUTOR_ADDED', 'EVENT_EXECUTOR_REMOVED',
+           'EVENT_JOBSTORE_ADDED', 'EVENT_JOBSTORE_REMOVED', 'EVENT_ALL_JOBS_REMOVED', 'EVENT_JOB_ADDED',
+           'EVENT_JOB_REMOVED', 'EVENT_JOB_MODIFIED', 'EVENT_JOB_EXECUTED', 'EVENT_JOB_ERROR', 'EVENT_JOB_MISSED',
+           'SchedulerEvent', 'JobEvent', 'JobExecutionEvent')
 
 
-EVENT_SCHEDULER_START = 1        # The scheduler was started
-EVENT_SCHEDULER_SHUTDOWN = 2     # The scheduler was shut down
-EVENT_JOBSTORE_ADDED = 4         # A job store was added to the scheduler
-EVENT_JOBSTORE_REMOVED = 8       # A job store was removed from the scheduler
-EVENT_JOBSTORE_JOB_ADDED = 16    # A job was added to a job store
-EVENT_JOBSTORE_JOB_REMOVED = 32  # A job was removed from a job store
-EVENT_JOB_EXECUTED = 64          # A job was executed successfully
-EVENT_JOB_ERROR = 128            # A job raised an exception during execution
-EVENT_JOB_MISSED = 256           # A job's execution was missed
-EVENT_ALL = (EVENT_SCHEDULER_START | EVENT_SCHEDULER_SHUTDOWN |
-             EVENT_JOBSTORE_ADDED | EVENT_JOBSTORE_REMOVED |
-             EVENT_JOBSTORE_JOB_ADDED | EVENT_JOBSTORE_JOB_REMOVED |
-             EVENT_JOB_EXECUTED | EVENT_JOB_ERROR | EVENT_JOB_MISSED)
+EVENT_SCHEDULER_START = 1
+EVENT_SCHEDULER_SHUTDOWN = 2
+EVENT_EXECUTOR_ADDED = 4
+EVENT_EXECUTOR_REMOVED = 8
+EVENT_JOBSTORE_ADDED = 16
+EVENT_JOBSTORE_REMOVED = 32
+EVENT_ALL_JOBS_REMOVED = 64
+EVENT_JOB_ADDED = 128
+EVENT_JOB_REMOVED = 256
+EVENT_JOB_MODIFIED = 512
+EVENT_JOB_EXECUTED = 1024
+EVENT_JOB_ERROR = 2048
+EVENT_JOB_MISSED = 4096
+EVENT_ALL = (EVENT_SCHEDULER_START | EVENT_SCHEDULER_SHUTDOWN | EVENT_JOBSTORE_ADDED | EVENT_JOBSTORE_REMOVED |
+             EVENT_JOB_ADDED | EVENT_JOB_REMOVED | EVENT_JOB_MODIFIED | EVENT_JOB_EXECUTED |
+             EVENT_JOB_ERROR | EVENT_JOB_MISSED)
 
 
 class SchedulerEvent(object):
     """
     An event that concerns the scheduler itself.
 
-    :var code: the type code of this event
+    :ivar code: the type code of this event
+    :ivar alias: alias of the job store or executor that was added or removed (if applicable)
     """
-    def __init__(self, code):
+
+    def __init__(self, code, alias=None):
+        super(SchedulerEvent, self).__init__()
         self.code = code
-
-
-class JobStoreEvent(SchedulerEvent):
-    """
-    An event that concerns job stores.
-
-    :var alias: the alias of the job store involved
-    :var job: the new job if a job was added
-    """
-    def __init__(self, code, alias, job=None):
-        SchedulerEvent.__init__(self, code)
         self.alias = alias
-        if job:
-            self.job = job
+
+    def __repr__(self):
+        return '<%s (code=%d)>' % (self.__class__.__name__, self.code)
 
 
 class JobEvent(SchedulerEvent):
+    """
+    An event that concerns a job.
+
+    :ivar code: the type code of this event
+    :ivar job_id: identifier of the job in question
+    :ivar jobstore: alias of the job store containing the job in question
+    """
+
+    def __init__(self, code, job_id, jobstore):
+        super(JobEvent, self).__init__(code)
+        self.code = code
+        self.job_id = job_id
+        self.jobstore = jobstore
+
+
+class JobExecutionEvent(JobEvent):
     """
     An event that concerns the execution of individual jobs.
 
-    :var job: the job instance in question
-    :var scheduled_run_time: the time when the job was scheduled to be run
-    :var retval: the return value of the successfully executed job
-    :var exception: the exception raised by the job
-    :var traceback: the traceback object associated with the exception
+    :ivar scheduled_run_time: the time when the job was scheduled to be run
+    :ivar retval: the return value of the successfully executed job
+    :ivar exception: the exception raised by the job
+    :ivar traceback: a formatted traceback for the exception
     """
-    def __init__(self, code, job, scheduled_run_time, retval=None,
-                 exception=None, traceback=None):
-        SchedulerEvent.__init__(self, code)
-        self.job = job
+
+    def __init__(self, code, job_id, jobstore, scheduled_run_time, retval=None, exception=None, traceback=None):
+        super(JobExecutionEvent, self).__init__(code, job_id, jobstore)
         self.scheduled_run_time = scheduled_run_time
         self.retval = retval
         self.exception = exception

lib/apscheduler/executors/asyncio.py

@@ -0,0 +1,28 @@
+from __future__ import absolute_import
+import sys
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class AsyncIOExecutor(BaseExecutor):
+    """
+    Runs jobs in the default executor of the event loop.
+
+    Plugin alias: ``asyncio``
+    """
+
+    def start(self, scheduler, alias):
+        super(AsyncIOExecutor, self).start(scheduler, alias)
+        self._eventloop = scheduler._eventloop
+
+    def _do_submit_job(self, job, run_times):
+        def callback(f):
+            try:
+                events = f.result()
+            except:
+                self._run_job_error(job.id, *sys.exc_info()[1:])
+            else:
+                self._run_job_success(job.id, events)
+
+        f = self._eventloop.run_in_executor(None, run_job, job, job._jobstore_alias, run_times, self._logger.name)
+        f.add_done_callback(callback)

lib/apscheduler/executors/base.py

@@ -0,0 +1,119 @@
+from abc import ABCMeta, abstractmethod
+from collections import defaultdict
+from datetime import datetime, timedelta
+from traceback import format_tb
+import logging
+import sys
+
+from pytz import utc
+import six
+
+from apscheduler.events import JobExecutionEvent, EVENT_JOB_MISSED, EVENT_JOB_ERROR, EVENT_JOB_EXECUTED
+
+
+class MaxInstancesReachedError(Exception):
+    def __init__(self, job):
+        super(MaxInstancesReachedError, self).__init__(
+            'Job "%s" has already reached its maximum number of instances (%d)' % (job.id, job.max_instances))
+
+
+class BaseExecutor(six.with_metaclass(ABCMeta, object)):
+    """Abstract base class that defines the interface that every executor must implement."""
+
+    _scheduler = None
+    _lock = None
+    _logger = logging.getLogger('apscheduler.executors')
+
+    def __init__(self):
+        super(BaseExecutor, self).__init__()
+        self._instances = defaultdict(lambda: 0)
+
+    def start(self, scheduler, alias):
+        """
+        Called by the scheduler when the scheduler is being started or when the executor is being added to an already
+        running scheduler.
+
+        :param apscheduler.schedulers.base.BaseScheduler scheduler: the scheduler that is starting this executor
+        :param str|unicode alias: alias of this executor as it was assigned to the scheduler
+        """
+
+        self._scheduler = scheduler
+        self._lock = scheduler._create_lock()
+        self._logger = logging.getLogger('apscheduler.executors.%s' % alias)
+
+    def shutdown(self, wait=True):
+        """
+        Shuts down this executor.
+
+        :param bool wait: ``True`` to wait until all submitted jobs have been executed
+        """
+
+    def submit_job(self, job, run_times):
+        """
+        Submits job for execution.
+
+        :param Job job: job to execute
+        :param list[datetime] run_times: list of datetimes specifying when the job should have been run
+        :raises MaxInstancesReachedError: if the maximum number of allowed instances for this job has been reached
+        """
+
+        assert self._lock is not None, 'This executor has not been started yet'
+        with self._lock:
+            if self._instances[job.id] >= job.max_instances:
+                raise MaxInstancesReachedError(job)
+
+            self._do_submit_job(job, run_times)
+            self._instances[job.id] += 1
+
+    @abstractmethod
+    def _do_submit_job(self, job, run_times):
+        """Performs the actual task of scheduling `run_job` to be called."""
+
+    def _run_job_success(self, job_id, events):
+        """Called by the executor with the list of generated events when `run_job` has been successfully called."""
+
+        with self._lock:
+            self._instances[job_id] -= 1
+
+        for event in events:
+            self._scheduler._dispatch_event(event)
+
+    def _run_job_error(self, job_id, exc, traceback=None):
+        """Called by the executor with the exception if there is an error calling `run_job`."""
+
+        with self._lock:
+            self._instances[job_id] -= 1
+
+        exc_info = (exc.__class__, exc, traceback)
+        self._logger.error('Error running job %s', job_id, exc_info=exc_info)
+
+
+def run_job(job, jobstore_alias, run_times, logger_name):
+    """Called by executors to run the job. Returns a list of scheduler events to be dispatched by the scheduler."""
+
+    events = []
+    logger = logging.getLogger(logger_name)
+    for run_time in run_times:
+        # See if the job missed its run time window, and handle possible misfires accordingly
+        if job.misfire_grace_time is not None:
+            difference = datetime.now(utc) - run_time
+            grace_time = timedelta(seconds=job.misfire_grace_time)
+            if difference > grace_time:
+                events.append(JobExecutionEvent(EVENT_JOB_MISSED, job.id, jobstore_alias, run_time))
+                logger.warning('Run time of job "%s" was missed by %s', job, difference)
+                continue
+
+        logger.info('Running job "%s" (scheduled at %s)', job, run_time)
+        try:
+            retval = job.func(*job.args, **job.kwargs)
+        except:
+            exc, tb = sys.exc_info()[1:]
+            formatted_tb = ''.join(format_tb(tb))
+            events.append(JobExecutionEvent(EVENT_JOB_ERROR, job.id, jobstore_alias, run_time, exception=exc,
+                                            traceback=formatted_tb))
+            logger.exception('Job "%s" raised an exception', job)
+        else:
+            events.append(JobExecutionEvent(EVENT_JOB_EXECUTED, job.id, jobstore_alias, run_time, retval=retval))
+            logger.info('Job "%s" executed successfully', job)
+
+    return events

lib/apscheduler/executors/debug.py

@@ -0,0 +1,19 @@
+import sys
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class DebugExecutor(BaseExecutor):
+    """
+    A special executor that executes the target callable directly instead of deferring it to a thread or process.
+
+    Plugin alias: ``debug``
+    """
+
+    def _do_submit_job(self, job, run_times):
+        try:
+            events = run_job(job, job._jobstore_alias, run_times, self._logger.name)
+        except:
+            self._run_job_error(job.id, *sys.exc_info()[1:])
+        else:
+            self._run_job_success(job.id, events)

lib/apscheduler/executors/gevent.py

@@ -0,0 +1,29 @@
+from __future__ import absolute_import
+import sys
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+try:
+    import gevent
+except ImportError:  # pragma: nocover
+    raise ImportError('GeventExecutor requires gevent installed')
+
+
+class GeventExecutor(BaseExecutor):
+    """
+    Runs jobs as greenlets.
+
+    Plugin alias: ``gevent``
+    """
+
+    def _do_submit_job(self, job, run_times):
+        def callback(greenlet):
+            try:
+                events = greenlet.get()
+            except:
+                self._run_job_error(job.id, *sys.exc_info()[1:])
+            else:
+                self._run_job_success(job.id, events)
+
+        gevent.spawn(run_job, job, job._jobstore_alias, run_times, self._logger.name).link(callback)

lib/apscheduler/executors/pool.py

@@ -0,0 +1,54 @@
+from abc import abstractmethod
+import concurrent.futures
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class BasePoolExecutor(BaseExecutor):
+    @abstractmethod
+    def __init__(self, pool):
+        super(BasePoolExecutor, self).__init__()
+        self._pool = pool
+
+    def _do_submit_job(self, job, run_times):
+        def callback(f):
+            exc, tb = (f.exception_info() if hasattr(f, 'exception_info') else
+                       (f.exception(), getattr(f.exception(), '__traceback__', None)))
+            if exc:
+                self._run_job_error(job.id, exc, tb)
+            else:
+                self._run_job_success(job.id, f.result())
+
+        f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
+        f.add_done_callback(callback)
+
+    def shutdown(self, wait=True):
+        self._pool.shutdown(wait)
+
+
+class ThreadPoolExecutor(BasePoolExecutor):
+    """
+    An executor that runs jobs in a concurrent.futures thread pool.
+
+    Plugin alias: ``threadpool``
+
+    :param max_workers: the maximum number of spawned threads.
+    """
+
+    def __init__(self, max_workers=10):
+        pool = concurrent.futures.ThreadPoolExecutor(int(max_workers))
+        super(ThreadPoolExecutor, self).__init__(pool)
+
+
+class ProcessPoolExecutor(BasePoolExecutor):
+    """
+    An executor that runs jobs in a concurrent.futures process pool.
+
+    Plugin alias: ``processpool``
+
+    :param max_workers: the maximum number of spawned processes.
+    """
+
+    def __init__(self, max_workers=10):
+        pool = concurrent.futures.ProcessPoolExecutor(int(max_workers))
+        super(ProcessPoolExecutor, self).__init__(pool)

lib/apscheduler/executors/twisted.py

@@ -0,0 +1,25 @@
+from __future__ import absolute_import
+
+from apscheduler.executors.base import BaseExecutor, run_job
+
+
+class TwistedExecutor(BaseExecutor):
+    """
+    Runs jobs in the reactor's thread pool.
+
+    Plugin alias: ``twisted``
+    """
+
+    def start(self, scheduler, alias):
+        super(TwistedExecutor, self).start(scheduler, alias)
+        self._reactor = scheduler._reactor
+
+    def _do_submit_job(self, job, run_times):
+        def callback(success, result):
+            if success:
+                self._run_job_success(job.id, result)
+            else:
+                self._run_job_error(job.id, result.value, result.tb)
+
+        self._reactor.getThreadPool().callInThreadWithCallback(callback, run_job, job, job._jobstore_alias, run_times,
+                                                               self._logger.name)

lib/apscheduler/job.py

@@ -1,134 +1,252 @@
-"""
-Jobs represent scheduled tasks.
-"""
+from collections import Iterable, Mapping
+from uuid import uuid4
 
-from threading import Lock
-from datetime import timedelta
+import six
 
-from apscheduler.util import to_unicode, ref_to_obj, get_callable_name,\
-    obj_to_ref
-
-
-class MaxInstancesReachedError(Exception):
-    pass
+from apscheduler.triggers.base import BaseTrigger
+from apscheduler.util import ref_to_obj, obj_to_ref, datetime_repr, repr_escape, get_callable_name, check_callable_args, \
+    convert_to_datetime
 
 
 class Job(object):
     """
-    Encapsulates the actual Job along with its metadata. Job instances
-    are created by the scheduler when adding jobs, and it should not be
-    directly instantiated.
+    Contains the options given when scheduling callables and its current schedule and other state.
+    This class should never be instantiated by the user.
 
-    :param trigger: trigger that determines the execution times
-    :param func: callable to call when the trigger is triggered
-    :param args: list of positional arguments to call func with
-    :param kwargs: dict of keyword arguments to call func with
-    :param name: name of the job (optional)
-    :param misfire_grace_time: seconds after the designated run time that
-        the job is still allowed to be run
-    :param coalesce: run once instead of many times if the scheduler determines
-        that the job should be run more than once in succession
-    :param max_runs: maximum number of times this job is allowed to be
-        triggered
-    :param max_instances: maximum number of concurrently running
-        instances allowed for this job
+    :var str id: the unique identifier of this job
+    :var str name: the description of this job
+    :var func: the callable to execute
+    :var tuple|list args: positional arguments to the callable
+    :var dict kwargs: keyword arguments to the callable
+    :var bool coalesce: whether to only run the job once when several run times are due
+    :var trigger: the trigger object that controls the schedule of this job
+    :var str executor: the name of the executor that will run this job
+    :var int misfire_grace_time: the time (in seconds) how much this job's execution is allowed to be late
+    :var int max_instances: the maximum number of concurrently executing instances allowed for this job
+    :var datetime.datetime next_run_time: the next scheduled run time of this job
     """
-    id = None
-    next_run_time = None
 
-    def __init__(self, trigger, func, args, kwargs, misfire_grace_time,
-                 coalesce, name=None, max_runs=None, max_instances=1):
-        if not trigger:
-            raise ValueError('The trigger must not be None')
-        if not hasattr(func, '__call__'):
-            raise TypeError('func must be callable')
-        if not hasattr(args, '__getitem__'):
-            raise TypeError('args must be a list-like object')
-        if not hasattr(kwargs, '__getitem__'):
-            raise TypeError('kwargs must be a dict-like object')
-        if misfire_grace_time <= 0:
-            raise ValueError('misfire_grace_time must be a positive value')
-        if max_runs is not None and max_runs <= 0:
-            raise ValueError('max_runs must be a positive value')
-        if max_instances <= 0:
-            raise ValueError('max_instances must be a positive value')
+    __slots__ = ('_scheduler', '_jobstore_alias', 'id', 'trigger', 'executor', 'func', 'func_ref', 'args', 'kwargs',
+                 'name', 'misfire_grace_time', 'coalesce', 'max_instances', 'next_run_time')
 
-        self._lock = Lock()
+    def __init__(self, scheduler, id=None, **kwargs):
+        super(Job, self).__init__()
+        self._scheduler = scheduler
+        self._jobstore_alias = None
+        self._modify(id=id or uuid4().hex, **kwargs)
 
-        self.trigger = trigger
-        self.func = func
-        self.args = args
-        self.kwargs = kwargs
-        self.name = to_unicode(name or get_callable_name(func))
-        self.misfire_grace_time = misfire_grace_time
-        self.coalesce = coalesce
-        self.max_runs = max_runs
-        self.max_instances = max_instances
-        self.runs = 0
-        self.instances = 0
-
-    def compute_next_run_time(self, now):
-        if self.runs == self.max_runs:
-            self.next_run_time = None
-        else:
-            self.next_run_time = self.trigger.get_next_fire_time(now)
-
-        return self.next_run_time
-
-    def get_run_times(self, now):
+    def modify(self, **changes):
         """
-        Computes the scheduled run times between ``next_run_time`` and ``now``.
+        Makes the given changes to this job and saves it in the associated job store.
+        Accepted keyword arguments are the same as the variables on this class.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.modify_job`
         """
+
+        self._scheduler.modify_job(self.id, self._jobstore_alias, **changes)
+
+    def reschedule(self, trigger, **trigger_args):
+        """
+        Shortcut for switching the trigger on this job.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.reschedule_job`
+        """
+
+        self._scheduler.reschedule_job(self.id, self._jobstore_alias, trigger, **trigger_args)
+
+    def pause(self):
+        """
+        Temporarily suspend the execution of this job.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.pause_job`
+        """
+
+        self._scheduler.pause_job(self.id, self._jobstore_alias)
+
+    def resume(self):
+        """
+        Resume the schedule of this job if previously paused.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.resume_job`
+        """
+
+        self._scheduler.resume_job(self.id, self._jobstore_alias)
+
+    def remove(self):
+        """
+        Unschedules this job and removes it from its associated job store.
+
+        .. seealso:: :meth:`~apscheduler.schedulers.base.BaseScheduler.remove_job`
+        """
+
+        self._scheduler.remove_job(self.id, self._jobstore_alias)
+
+    @property
+    def pending(self):
+        """Returns ``True`` if the referenced job is still waiting to be added to its designated job store."""
+
+        return self._jobstore_alias is None
+
+    #
+    # Private API
+    #
+
+    def _get_run_times(self, now):
+        """
+        Computes the scheduled run times between ``next_run_time`` and ``now`` (inclusive).
+
+        :type now: datetime.datetime
+        :rtype: list[datetime.datetime]
+        """
+
         run_times = []
-        run_time = self.next_run_time
-        increment = timedelta(microseconds=1)
-        while ((not self.max_runs or self.runs < self.max_runs) and
-               run_time and run_time <= now):
-            run_times.append(run_time)
-            run_time = self.trigger.get_next_fire_time(run_time + increment)
+        next_run_time = self.next_run_time
+        while next_run_time and next_run_time <= now:
+            run_times.append(next_run_time)
+            next_run_time = self.trigger.get_next_fire_time(next_run_time, now)
 
         return run_times
 
-    def add_instance(self):
-        self._lock.acquire()
-        try:
-            if self.instances == self.max_instances:
-                raise MaxInstancesReachedError
-            self.instances += 1
-        finally:
-            self._lock.release()
+    def _modify(self, **changes):
+        """Validates the changes to the Job and makes the modifications if and only if all of them validate."""
 
-    def remove_instance(self):
-        self._lock.acquire()
-        try:
-            assert self.instances > 0, 'Already at 0 instances'
-            self.instances -= 1
-        finally:
-            self._lock.release()
+        approved = {}
+
+        if 'id' in changes:
+            value = changes.pop('id')
+            if not isinstance(value, six.string_types):
+                raise TypeError("id must be a nonempty string")
+            if hasattr(self, 'id'):
+                raise ValueError('The job ID may not be changed')
+            approved['id'] = value
+
+        if 'func' in changes or 'args' in changes or 'kwargs' in changes:
+            func = changes.pop('func') if 'func' in changes else self.func
+            args = changes.pop('args') if 'args' in changes else self.args
+            kwargs = changes.pop('kwargs') if 'kwargs' in changes else self.kwargs
+
+            if isinstance(func, str):
+                func_ref = func
+                func = ref_to_obj(func)
+            elif callable(func):
+                try:
+                    func_ref = obj_to_ref(func)
+                except ValueError:
+                    # If this happens, this Job won't be serializable
+                    func_ref = None
+            else:
+                raise TypeError('func must be a callable or a textual reference to one')
+
+            if not hasattr(self, 'name') and changes.get('name', None) is None:
+                changes['name'] = get_callable_name(func)
+
+            if isinstance(args, six.string_types) or not isinstance(args, Iterable):
+                raise TypeError('args must be a non-string iterable')
+            if isinstance(kwargs, six.string_types) or not isinstance(kwargs, Mapping):
+                raise TypeError('kwargs must be a dict-like object')
+
+            check_callable_args(func, args, kwargs)
+
+            approved['func'] = func
+            approved['func_ref'] = func_ref
+            approved['args'] = args
+            approved['kwargs'] = kwargs
+
+        if 'name' in changes:
+            value = changes.pop('name')
+            if not value or not isinstance(value, six.string_types):
+                raise TypeError("name must be a nonempty string")
+            approved['name'] = value
+
+        if 'misfire_grace_time' in changes:
+            value = changes.pop('misfire_grace_time')
+            if value is not None and (not isinstance(value, six.integer_types) or value <= 0):
+                raise TypeError('misfire_grace_time must be either None or a positive integer')
+            approved['misfire_grace_time'] = value
+
+        if 'coalesce' in changes:
+            value = bool(changes.pop('coalesce'))
+            approved['coalesce'] = value
+
+        if 'max_instances' in changes:
+            value = changes.pop('max_instances')
+            if not isinstance(value, six.integer_types) or value <= 0:
+                raise TypeError('max_instances must be a positive integer')
+            approved['max_instances'] = value
+
+        if 'trigger' in changes:
+            trigger = changes.pop('trigger')
+            if not isinstance(trigger, BaseTrigger):
+                raise TypeError('Expected a trigger instance, got %s instead' % trigger.__class__.__name__)
+
+            approved['trigger'] = trigger
+
+        if 'executor' in changes:
+            value = changes.pop('executor')
+            if not isinstance(value, six.string_types):
+                raise TypeError('executor must be a string')
+            approved['executor'] = value
+
+        if 'next_run_time' in changes:
+            value = changes.pop('next_run_time')
+            approved['next_run_time'] = convert_to_datetime(value, self._scheduler.timezone, 'next_run_time')
+
+        if changes:
+            raise AttributeError('The following are not modifiable attributes of Job: %s' % ', '.join(changes))
+
+        for key, value in six.iteritems(approved):
+            setattr(self, key, value)
 
     def __getstate__(self):
-        # Prevents the unwanted pickling of transient or unpicklable variables
-        state = self.__dict__.copy()
-        state.pop('instances', None)
-        state.pop('func', None)
-        state.pop('_lock', None)
-        state['func_ref'] = obj_to_ref(self.func)
-        return state
+        # Don't allow this Job to be serialized if the function reference could not be determined
+        if not self.func_ref:
+            raise ValueError('This Job cannot be serialized since the reference to its callable (%r) could not be '
+                             'determined. Consider giving a textual reference (module:function name) instead.' %
+                             (self.func,))
+
+        return {
+            'version': 1,
+            'id': self.id,
+            'func': self.func_ref,
+            'trigger': self.trigger,
+            'executor': self.executor,
+            'args': self.args,
+            'kwargs': self.kwargs,
+            'name': self.name,
+            'misfire_grace_time': self.misfire_grace_time,
+            'coalesce': self.coalesce,
+            'max_instances': self.max_instances,
+            'next_run_time': self.next_run_time
+        }
 
     def __setstate__(self, state):
-        state['instances'] = 0
-        state['func'] = ref_to_obj(state.pop('func_ref'))
-        state['_lock'] = Lock()
-        self.__dict__ = state
+        if state.get('version', 1) > 1:
+            raise ValueError('Job has version %s, but only version 1 can be handled' % state['version'])
+
+        self.id = state['id']
+        self.func_ref = state['func']
+        self.func = ref_to_obj(self.func_ref)
+        self.trigger = state['trigger']
+        self.executor = state['executor']
+        self.args = state['args']
+        self.kwargs = state['kwargs']
+        self.name = state['name']
+        self.misfire_grace_time = state['misfire_grace_time']
+        self.coalesce = state['coalesce']
+        self.max_instances = state['max_instances']
+        self.next_run_time = state['next_run_time']
 
     def __eq__(self, other):
         if isinstance(other, Job):
-            return self.id is not None and other.id == self.id or self is other
+            return self.id == other.id
         return NotImplemented
 
     def __repr__(self):
-        return '<Job (name=%s, trigger=%s)>' % (self.name, repr(self.trigger))
+        return '<Job (id=%s name=%s)>' % (repr_escape(self.id), repr_escape(self.name))
 
     def __str__(self):
-        return '%s (trigger: %s, next run at: %s)' % (self.name,
-            str(self.trigger), str(self.next_run_time))
+        return '%s (trigger: %s, next run at: %s)' % (repr_escape(self.name), repr_escape(str(self.trigger)),
+                                                      datetime_repr(self.next_run_time))
+
+    def __unicode__(self):
+        return six.u('%s (trigger: %s, next run at: %s)') % (self.name, self.trigger, datetime_repr(self.next_run_time))

lib/apscheduler/jobstores/base.py

@@ -1,25 +1,127 @@
-"""
-Abstract base class that provides the interface needed by all job stores.
-Job store methods are also documented here.
-"""
+from abc import ABCMeta, abstractmethod
+import logging
+
+import six
 
 
-class JobStore(object):
-    def add_job(self, job):
-        """Adds the given job from this store."""
-        raise NotImplementedError
+class JobLookupError(KeyError):
+    """Raised when the job store cannot find a job for update or removal."""
 
-    def update_job(self, job):
-        """Persists the running state of the given job."""
-        raise NotImplementedError
+    def __init__(self, job_id):
+        super(JobLookupError, self).__init__(six.u('No job by the id of %s was found') % job_id)
 
-    def remove_job(self, job):
-        """Removes the given jobs from this store."""
-        raise NotImplementedError
 
-    def load_jobs(self):
-        """Loads jobs from this store into memory."""
-        raise NotImplementedError
+class ConflictingIdError(KeyError):
+    """Raised when the uniqueness of job IDs is being violated."""
 
-    def close(self):
+    def __init__(self, job_id):
+        super(ConflictingIdError, self).__init__(six.u('Job identifier (%s) conflicts with an existing job') % job_id)
+
+
+class TransientJobError(ValueError):
+    """Raised when an attempt to add transient (with no func_ref) job to a persistent job store is detected."""
+
+    def __init__(self, job_id):
+        super(TransientJobError, self).__init__(
+            six.u('Job (%s) cannot be added to this job store because a reference to the callable could not be '
+                  'determined.') % job_id)
+
+
+class BaseJobStore(six.with_metaclass(ABCMeta)):
+    """Abstract base class that defines the interface that every job store must implement."""
+
+    _scheduler = None
+    _alias = None
+    _logger = logging.getLogger('apscheduler.jobstores')
+
+    def start(self, scheduler, alias):
+        """
+        Called by the scheduler when the scheduler is being started or when the job store is being added to an already
+        running scheduler.
+
+        :param apscheduler.schedulers.base.BaseScheduler scheduler: the scheduler that is starting this job store
+        :param str|unicode alias: alias of this job store as it was assigned to the scheduler
+        """
+
+        self._scheduler = scheduler
+        self._alias = alias
+        self._logger = logging.getLogger('apscheduler.jobstores.%s' % alias)
+
+    def shutdown(self):
         """Frees any resources still bound to this job store."""
+
+    @abstractmethod
+    def lookup_job(self, job_id):
+        """
+        Returns a specific job, or ``None`` if it isn't found..
+
+        The job store is responsible for setting the ``scheduler`` and ``jobstore`` attributes of the returned job to
+        point to the scheduler and itself, respectively.
+
+        :param str|unicode job_id: identifier of the job
+        :rtype: Job
+        """
+
+    @abstractmethod
+    def get_due_jobs(self, now):
+        """
+        Returns the list of jobs that have ``next_run_time`` earlier or equal to ``now``.
+        The returned jobs must be sorted by next run time (ascending).
+
+        :param datetime.datetime now: the current (timezone aware) datetime
+        :rtype: list[Job]
+        """
+
+    @abstractmethod
+    def get_next_run_time(self):
+        """
+        Returns the earliest run time of all the jobs stored in this job store, or ``None`` if there are no active jobs.
+
+        :rtype: datetime.datetime
+        """
+
+    @abstractmethod
+    def get_all_jobs(self):
+        """
+        Returns a list of all jobs in this job store. The returned jobs should be sorted by next run time (ascending).
+        Paused jobs (next_run_time is None) should be sorted last.
+
+        The job store is responsible for setting the ``scheduler`` and ``jobstore`` attributes of the returned jobs to
+        point to the scheduler and itself, respectively.
+
+        :rtype: list[Job]
+        """
+
+    @abstractmethod
+    def add_job(self, job):
+        """
+        Adds the given job to this store.
+
+        :param Job job: the job to add
+        :raises ConflictingIdError: if there is another job in this store with the same ID
+        """
+
+    @abstractmethod
+    def update_job(self, job):
+        """
+        Replaces the job in the store with the given newer version.
+
+        :param Job job: the job to update
+        :raises JobLookupError: if the job does not exist
+        """
+
+    @abstractmethod
+    def remove_job(self, job_id):
+        """
+        Removes the given job from this store.
+
+        :param str|unicode job_id: identifier of the job
+        :raises JobLookupError: if the job does not exist
+        """
+
+    @abstractmethod
+    def remove_all_jobs(self):
+        """Removes all jobs from this store."""
+
+    def __repr__(self):
+        return '<%s>' % self.__class__.__name__

lib/apscheduler/jobstores/memory.py

@@ -0,0 +1,107 @@
+from __future__ import absolute_import
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import datetime_to_utc_timestamp
+
+
+class MemoryJobStore(BaseJobStore):
+    """
+    Stores jobs in an array in RAM. Provides no persistence support.
+
+    Plugin alias: ``memory``
+    """
+
+    def __init__(self):
+        super(MemoryJobStore, self).__init__()
+        self._jobs = []  # list of (job, timestamp), sorted by next_run_time and job id (ascending)
+        self._jobs_index = {}  # id -> (job, timestamp) lookup table
+
+    def lookup_job(self, job_id):
+        return self._jobs_index.get(job_id, (None, None))[0]
+
+    def get_due_jobs(self, now):
+        now_timestamp = datetime_to_utc_timestamp(now)
+        pending = []
+        for job, timestamp in self._jobs:
+            if timestamp is None or timestamp > now_timestamp:
+                break
+            pending.append(job)
+
+        return pending
+
+    def get_next_run_time(self):
+        return self._jobs[0][0].next_run_time if self._jobs else None
+
+    def get_all_jobs(self):
+        return [j[0] for j in self._jobs]
+
+    def add_job(self, job):
+        if job.id in self._jobs_index:
+            raise ConflictingIdError(job.id)
+
+        timestamp = datetime_to_utc_timestamp(job.next_run_time)
+        index = self._get_job_index(timestamp, job.id)
+        self._jobs.insert(index, (job, timestamp))
+        self._jobs_index[job.id] = (job, timestamp)
+
+    def update_job(self, job):
+        old_job, old_timestamp = self._jobs_index.get(job.id, (None, None))
+        if old_job is None:
+            raise JobLookupError(job.id)
+
+        # If the next run time has not changed, simply replace the job in its present index.
+        # Otherwise, reinsert the job to the list to preserve the ordering.
+        old_index = self._get_job_index(old_timestamp, old_job.id)
+        new_timestamp = datetime_to_utc_timestamp(job.next_run_time)
+        if old_timestamp == new_timestamp:
+            self._jobs[old_index] = (job, new_timestamp)
+        else:
+            del self._jobs[old_index]
+            new_index = self._get_job_index(new_timestamp, job.id)
+            self._jobs.insert(new_index, (job, new_timestamp))
+
+        self._jobs_index[old_job.id] = (job, new_timestamp)
+
+    def remove_job(self, job_id):
+        job, timestamp = self._jobs_index.get(job_id, (None, None))
+        if job is None:
+            raise JobLookupError(job_id)
+
+        index = self._get_job_index(timestamp, job_id)
+        del self._jobs[index]
+        del self._jobs_index[job.id]
+
+    def remove_all_jobs(self):
+        self._jobs = []
+        self._jobs_index = {}
+
+    def shutdown(self):
+        self.remove_all_jobs()
+
+    def _get_job_index(self, timestamp, job_id):
+        """
+        Returns the index of the given job, or if it's not found, the index where the job should be inserted based on
+        the given timestamp.
+
+        :type timestamp: int
+        :type job_id: str
+        """
+
+        lo, hi = 0, len(self._jobs)
+        timestamp = float('inf') if timestamp is None else timestamp
+        while lo < hi:
+            mid = (lo + hi) // 2
+            mid_job, mid_timestamp = self._jobs[mid]
+            mid_timestamp = float('inf') if mid_timestamp is None else mid_timestamp
+            if mid_timestamp > timestamp:
+                hi = mid
+            elif mid_timestamp < timestamp:
+                lo = mid + 1
+            elif mid_job.id > job_id:
+                hi = mid
+            elif mid_job.id < job_id:
+                lo = mid + 1
+            else:
+                return mid
+
+        return lo

lib/apscheduler/jobstores/mongodb.py

@@ -0,0 +1,124 @@
+from __future__ import absolute_import
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import maybe_ref, datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from bson.binary import Binary
+    from pymongo.errors import DuplicateKeyError
+    from pymongo import MongoClient, ASCENDING
+except ImportError:  # pragma: nocover
+    raise ImportError('MongoDBJobStore requires PyMongo installed')
+
+
+class MongoDBJobStore(BaseJobStore):
+    """
+    Stores jobs in a MongoDB database. Any leftover keyword arguments are directly passed to pymongo's `MongoClient
+    <http://api.mongodb.org/python/current/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient>`_.
+
+    Plugin alias: ``mongodb``
+
+    :param str database: database to store jobs in
+    :param str collection: collection to store jobs in
+    :param client: a :class:`~pymongo.mongo_client.MongoClient` instance to use instead of providing connection
+                   arguments
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    """
+
+    def __init__(self, database='apscheduler', collection='jobs', client=None,
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL, **connect_args):
+        super(MongoDBJobStore, self).__init__()
+        self.pickle_protocol = pickle_protocol
+
+        if not database:
+            raise ValueError('The "database" parameter must not be empty')
+        if not collection:
+            raise ValueError('The "collection" parameter must not be empty')
+
+        if client:
+            self.connection = maybe_ref(client)
+        else:
+            connect_args.setdefault('w', 1)
+            self.connection = MongoClient(**connect_args)
+
+        self.collection = self.connection[database][collection]
+        self.collection.ensure_index('next_run_time', sparse=True)
+
+    def lookup_job(self, job_id):
+        document = self.collection.find_one(job_id, ['job_state'])
+        return self._reconstitute_job(document['job_state']) if document else None
+
+    def get_due_jobs(self, now):
+        timestamp = datetime_to_utc_timestamp(now)
+        return self._get_jobs({'next_run_time': {'$lte': timestamp}})
+
+    def get_next_run_time(self):
+        document = self.collection.find_one({'next_run_time': {'$ne': None}}, fields=['next_run_time'],
+                                            sort=[('next_run_time', ASCENDING)])
+        return utc_timestamp_to_datetime(document['next_run_time']) if document else None
+
+    def get_all_jobs(self):
+        return self._get_jobs({})
+
+    def add_job(self, job):
+        try:
+            self.collection.insert({
+                '_id': job.id,
+                'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+                'job_state': Binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
+            })
+        except DuplicateKeyError:
+            raise ConflictingIdError(job.id)
+
+    def update_job(self, job):
+        changes = {
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': Binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
+        }
+        result = self.collection.update({'_id': job.id}, {'$set': changes})
+        if result and result['n'] == 0:
+            raise JobLookupError(id)
+
+    def remove_job(self, job_id):
+        result = self.collection.remove(job_id)
+        if result and result['n'] == 0:
+            raise JobLookupError(job_id)
+
+    def remove_all_jobs(self):
+        self.collection.remove()
+
+    def shutdown(self):
+        self.connection.disconnect()
+
+    def _reconstitute_job(self, job_state):
+        job_state = pickle.loads(job_state)
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _get_jobs(self, conditions):
+        jobs = []
+        failed_job_ids = []
+        for document in self.collection.find(conditions, ['_id', 'job_state'], sort=[('next_run_time', ASCENDING)]):
+            try:
+                jobs.append(self._reconstitute_job(document['job_state']))
+            except:
+                self._logger.exception('Unable to restore job "%s" -- removing it', document['_id'])
+                failed_job_ids.append(document['_id'])
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            self.collection.remove({'_id': {'$in': failed_job_ids}})
+
+        return jobs
+
+    def __repr__(self):
+        return '<%s (client=%s)>' % (self.__class__.__name__, self.connection)

lib/apscheduler/jobstores/mongodb_store.py

@@ -1,84 +0,0 @@
-"""
-Stores jobs in a MongoDB database.
-"""
-import logging
-
-from apscheduler.jobstores.base import JobStore
-from apscheduler.job import Job
-
-try:
-    import cPickle as pickle
-except ImportError:  # pragma: nocover
-    import pickle
-
-try:
-    from bson.binary import Binary
-    from pymongo.connection import Connection
-except ImportError:  # pragma: nocover
-    raise ImportError('MongoDBJobStore requires PyMongo installed')
-
-logger = logging.getLogger(__name__)
-
-
-class MongoDBJobStore(JobStore):
-    def __init__(self, database='apscheduler', collection='jobs',
-                 connection=None, pickle_protocol=pickle.HIGHEST_PROTOCOL,
-                 **connect_args):
-        self.jobs = []
-        self.pickle_protocol = pickle_protocol
-
-        if not database:
-            raise ValueError('The "database" parameter must not be empty')
-        if not collection:
-            raise ValueError('The "collection" parameter must not be empty')
-
-        if connection:
-            self.connection = connection
-        else:
-            self.connection = Connection(**connect_args)
-
-        self.collection = self.connection[database][collection]
-
-    def add_job(self, job):
-        job_dict = job.__getstate__()
-        job_dict['trigger'] = Binary(pickle.dumps(job.trigger,
-                                                  self.pickle_protocol))
-        job_dict['args'] = Binary(pickle.dumps(job.args,
-                                               self.pickle_protocol))
-        job_dict['kwargs'] = Binary(pickle.dumps(job.kwargs,
-                                                 self.pickle_protocol))
-        job.id = self.collection.insert(job_dict)
-        self.jobs.append(job)
-
-    def remove_job(self, job):
-        self.collection.remove(job.id)
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        jobs = []
-        for job_dict in self.collection.find():
-            try:
-                job = Job.__new__(Job)
-                job_dict['id'] = job_dict.pop('_id')
-                job_dict['trigger'] = pickle.loads(job_dict['trigger'])
-                job_dict['args'] = pickle.loads(job_dict['args'])
-                job_dict['kwargs'] = pickle.loads(job_dict['kwargs'])
-                job.__setstate__(job_dict)
-                jobs.append(job)
-            except Exception:
-                job_name = job_dict.get('name', '(unknown)')
-                logger.exception('Unable to restore job "%s"', job_name)
-        self.jobs = jobs
-
-    def update_job(self, job):
-        spec = {'_id': job.id}
-        document = {'$set': {'next_run_time': job.next_run_time},
-                    '$inc': {'runs': 1}}
-        self.collection.update(spec, document)
-
-    def close(self):
-        self.connection.disconnect()
-
-    def __repr__(self):
-        connection = self.collection.database.connection
-        return '<%s (connection=%s)>' % (self.__class__.__name__, connection)

lib/apscheduler/jobstores/ram_store.py

@@ -1,25 +0,0 @@
-"""
-Stores jobs in an array in RAM. Provides no persistence support.
-"""
-
-from apscheduler.jobstores.base import JobStore
-
-
-class RAMJobStore(JobStore):
-    def __init__(self):
-        self.jobs = []
-
-    def add_job(self, job):
-        self.jobs.append(job)
-
-    def update_job(self, job):
-        pass
-
-    def remove_job(self, job):
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        pass
-
-    def __repr__(self):
-        return '<%s>' % (self.__class__.__name__)

lib/apscheduler/jobstores/redis.py

@@ -0,0 +1,138 @@
+from __future__ import absolute_import
+
+import six
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from redis import StrictRedis
+except ImportError:  # pragma: nocover
+    raise ImportError('RedisJobStore requires redis installed')
+
+
+class RedisJobStore(BaseJobStore):
+    """
+    Stores jobs in a Redis database. Any leftover keyword arguments are directly passed to redis's StrictRedis.
+
+    Plugin alias: ``redis``
+
+    :param int db: the database number to store jobs in
+    :param str jobs_key: key to store jobs in
+    :param str run_times_key: key to store the jobs' run times in
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    """
+
+    def __init__(self, db=0, jobs_key='apscheduler.jobs', run_times_key='apscheduler.run_times',
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL, **connect_args):
+        super(RedisJobStore, self).__init__()
+
+        if db is None:
+            raise ValueError('The "db" parameter must not be empty')
+        if not jobs_key:
+            raise ValueError('The "jobs_key" parameter must not be empty')
+        if not run_times_key:
+            raise ValueError('The "run_times_key" parameter must not be empty')
+
+        self.pickle_protocol = pickle_protocol
+        self.jobs_key = jobs_key
+        self.run_times_key = run_times_key
+        self.redis = StrictRedis(db=int(db), **connect_args)
+
+    def lookup_job(self, job_id):
+        job_state = self.redis.hget(self.jobs_key, job_id)
+        return self._reconstitute_job(job_state) if job_state else None
+
+    def get_due_jobs(self, now):
+        timestamp = datetime_to_utc_timestamp(now)
+        job_ids = self.redis.zrangebyscore(self.run_times_key, 0, timestamp)
+        if job_ids:
+            job_states = self.redis.hmget(self.jobs_key, *job_ids)
+            return self._reconstitute_jobs(six.moves.zip(job_ids, job_states))
+        return []
+
+    def get_next_run_time(self):
+        next_run_time = self.redis.zrange(self.run_times_key, 0, 0, withscores=True)
+        if next_run_time:
+            return utc_timestamp_to_datetime(next_run_time[0][1])
+
+    def get_all_jobs(self):
+        job_states = self.redis.hgetall(self.jobs_key)
+        jobs = self._reconstitute_jobs(six.iteritems(job_states))
+        return sorted(jobs, key=lambda job: job.next_run_time)
+
+    def add_job(self, job):
+        if self.redis.hexists(self.jobs_key, job.id):
+            raise ConflictingIdError(job.id)
+
+        with self.redis.pipeline() as pipe:
+            pipe.multi()
+            pipe.hset(self.jobs_key, job.id, pickle.dumps(job.__getstate__(), self.pickle_protocol))
+            pipe.zadd(self.run_times_key, datetime_to_utc_timestamp(job.next_run_time), job.id)
+            pipe.execute()
+
+    def update_job(self, job):
+        if not self.redis.hexists(self.jobs_key, job.id):
+            raise JobLookupError(job.id)
+
+        with self.redis.pipeline() as pipe:
+            pipe.hset(self.jobs_key, job.id, pickle.dumps(job.__getstate__(), self.pickle_protocol))
+            if job.next_run_time:
+                pipe.zadd(self.run_times_key, datetime_to_utc_timestamp(job.next_run_time), job.id)
+            else:
+                pipe.zrem(self.run_times_key, job.id)
+            pipe.execute()
+
+    def remove_job(self, job_id):
+        if not self.redis.hexists(self.jobs_key, job_id):
+            raise JobLookupError(job_id)
+
+        with self.redis.pipeline() as pipe:
+            pipe.hdel(self.jobs_key, job_id)
+            pipe.zrem(self.run_times_key, job_id)
+            pipe.execute()
+
+    def remove_all_jobs(self):
+        with self.redis.pipeline() as pipe:
+            pipe.delete(self.jobs_key)
+            pipe.delete(self.run_times_key)
+            pipe.execute()
+
+    def shutdown(self):
+        self.redis.connection_pool.disconnect()
+
+    def _reconstitute_job(self, job_state):
+        job_state = pickle.loads(job_state)
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _reconstitute_jobs(self, job_states):
+        jobs = []
+        failed_job_ids = []
+        for job_id, job_state in job_states:
+            try:
+                jobs.append(self._reconstitute_job(job_state))
+            except:
+                self._logger.exception('Unable to restore job "%s" -- removing it', job_id)
+                failed_job_ids.append(job_id)
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            with self.redis.pipeline() as pipe:
+                pipe.hdel(self.jobs_key, *failed_job_ids)
+                pipe.zrem(self.run_times_key, *failed_job_ids)
+                pipe.execute()
+
+        return jobs
+
+    def __repr__(self):
+        return '<%s>' % self.__class__.__name__

lib/apscheduler/jobstores/shelve_store.py

@@ -1,65 +0,0 @@
-"""
-Stores jobs in a file governed by the :mod:`shelve` module.
-"""
-
-import shelve
-import pickle
-import random
-import logging
-
-from apscheduler.jobstores.base import JobStore
-from apscheduler.job import Job
-from apscheduler.util import itervalues
-
-logger = logging.getLogger(__name__)
-
-
-class ShelveJobStore(JobStore):
-    MAX_ID = 1000000
-
-    def __init__(self, path, pickle_protocol=pickle.HIGHEST_PROTOCOL):
-        self.jobs = []
-        self.path = path
-        self.pickle_protocol = pickle_protocol
-        self.store = shelve.open(path, 'c', self.pickle_protocol)
-
-    def _generate_id(self):
-        id = None
-        while not id:
-            id = str(random.randint(1, self.MAX_ID))
-            if not id in self.store:
-                return id
-
-    def add_job(self, job):
-        job.id = self._generate_id()
-        self.jobs.append(job)
-        self.store[job.id] = job.__getstate__()
-
-    def update_job(self, job):
-        job_dict = self.store[job.id]
-        job_dict['next_run_time'] = job.next_run_time
-        job_dict['runs'] = job.runs
-        self.store[job.id] = job_dict
-
-    def remove_job(self, job):
-        del self.store[job.id]
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        jobs = []
-        for job_dict in itervalues(self.store):
-            try:
-                job = Job.__new__(Job)
-                job.__setstate__(job_dict)
-                jobs.append(job)
-            except Exception:
-                job_name = job_dict.get('name', '(unknown)')
-                logger.exception('Unable to restore job "%s"', job_name)
-
-        self.jobs = jobs
-
-    def close(self):
-        self.store.close()
-
-    def __repr__(self):
-        return '<%s (path=%s)>' % (self.__class__.__name__, self.path)

lib/apscheduler/jobstores/sqlalchemy.py

@@ -0,0 +1,137 @@
+from __future__ import absolute_import
+
+from apscheduler.jobstores.base import BaseJobStore, JobLookupError, ConflictingIdError
+from apscheduler.util import maybe_ref, datetime_to_utc_timestamp, utc_timestamp_to_datetime
+from apscheduler.job import Job
+
+try:
+    import cPickle as pickle
+except ImportError:  # pragma: nocover
+    import pickle
+
+try:
+    from sqlalchemy import create_engine, Table, Column, MetaData, Unicode, Float, LargeBinary, select
+    from sqlalchemy.exc import IntegrityError
+except ImportError:  # pragma: nocover
+    raise ImportError('SQLAlchemyJobStore requires SQLAlchemy installed')
+
+
+class SQLAlchemyJobStore(BaseJobStore):
+    """
+    Stores jobs in a database table using SQLAlchemy. The table will be created if it doesn't exist in the database.
+
+    Plugin alias: ``sqlalchemy``
+
+    :param str url: connection string (see `SQLAlchemy documentation
+                    <http://docs.sqlalchemy.org/en/latest/core/engines.html?highlight=create_engine#database-urls>`_
+                    on this)
+    :param engine: an SQLAlchemy Engine to use instead of creating a new one based on ``url``
+    :param str tablename: name of the table to store jobs in
+    :param metadata: a :class:`~sqlalchemy.MetaData` instance to use instead of creating a new one
+    :param int pickle_protocol: pickle protocol level to use (for serialization), defaults to the highest available
+    """
+
+    def __init__(self, url=None, engine=None, tablename='apscheduler_jobs', metadata=None,
+                 pickle_protocol=pickle.HIGHEST_PROTOCOL):
+        super(SQLAlchemyJobStore, self).__init__()
+        self.pickle_protocol = pickle_protocol
+        metadata = maybe_ref(metadata) or MetaData()
+
+        if engine:
+            self.engine = maybe_ref(engine)
+        elif url:
+            self.engine = create_engine(url)
+        else:
+            raise ValueError('Need either "engine" or "url" defined')
+
+        # 191 = max key length in MySQL for InnoDB/utf8mb4 tables, 25 = precision that translates to an 8-byte float
+        self.jobs_t = Table(
+            tablename, metadata,
+            Column('id', Unicode(191, _warn_on_bytestring=False), primary_key=True),
+            Column('next_run_time', Float(25), index=True),
+            Column('job_state', LargeBinary, nullable=False)
+        )
+
+        self.jobs_t.create(self.engine, True)
+
+    def lookup_job(self, job_id):
+        selectable = select([self.jobs_t.c.job_state]).where(self.jobs_t.c.id == job_id)
+        job_state = self.engine.execute(selectable).scalar()
+        return self._reconstitute_job(job_state) if job_state else None
+
+    def get_due_jobs(self, now):
+        timestamp = datetime_to_utc_timestamp(now)
+        return self._get_jobs(self.jobs_t.c.next_run_time <= timestamp)
+
+    def get_next_run_time(self):
+        selectable = select([self.jobs_t.c.next_run_time]).where(self.jobs_t.c.next_run_time != None).\
+            order_by(self.jobs_t.c.next_run_time).limit(1)
+        next_run_time = self.engine.execute(selectable).scalar()
+        return utc_timestamp_to_datetime(next_run_time)
+
+    def get_all_jobs(self):
+        return self._get_jobs()
+
+    def add_job(self, job):
+        insert = self.jobs_t.insert().values(**{
+            'id': job.id,
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': pickle.dumps(job.__getstate__(), self.pickle_protocol)
+        })
+        try:
+            self.engine.execute(insert)
+        except IntegrityError:
+            raise ConflictingIdError(job.id)
+
+    def update_job(self, job):
+        update = self.jobs_t.update().values(**{
+            'next_run_time': datetime_to_utc_timestamp(job.next_run_time),
+            'job_state': pickle.dumps(job.__getstate__(), self.pickle_protocol)
+        }).where(self.jobs_t.c.id == job.id)
+        result = self.engine.execute(update)
+        if result.rowcount == 0:
+            raise JobLookupError(id)
+
+    def remove_job(self, job_id):
+        delete = self.jobs_t.delete().where(self.jobs_t.c.id == job_id)
+        result = self.engine.execute(delete)
+        if result.rowcount == 0:
+            raise JobLookupError(job_id)
+
+    def remove_all_jobs(self):
+        delete = self.jobs_t.delete()
+        self.engine.execute(delete)
+
+    def shutdown(self):
+        self.engine.dispose()
+
+    def _reconstitute_job(self, job_state):
+        job_state = pickle.loads(job_state)
+        job_state['jobstore'] = self
+        job = Job.__new__(Job)
+        job.__setstate__(job_state)
+        job._scheduler = self._scheduler
+        job._jobstore_alias = self._alias
+        return job
+
+    def _get_jobs(self, *conditions):
+        jobs = []
+        selectable = select([self.jobs_t.c.id, self.jobs_t.c.job_state]).order_by(self.jobs_t.c.next_run_time)
+        selectable = selectable.where(*conditions) if conditions else selectable
+        failed_job_ids = set()
+        for row in self.engine.execute(selectable):
+            try:
+                jobs.append(self._reconstitute_job(row.job_state))
+            except:
+                self._logger.exception('Unable to restore job "%s" -- removing it', row.id)
+                failed_job_ids.add(row.id)
+
+        # Remove all the jobs we failed to restore
+        if failed_job_ids:
+            delete = self.jobs_t.delete().where(self.jobs_t.c.id.in_(failed_job_ids))
+            self.engine.execute(delete)
+
+        return jobs
+
+    def __repr__(self):
+        return '<%s (url=%s)>' % (self.__class__.__name__, self.engine.url)

lib/apscheduler/jobstores/sqlalchemy_store.py

@@ -1,87 +0,0 @@
-"""
-Stores jobs in a database table using SQLAlchemy.
-"""
-import pickle
-import logging
-
-from apscheduler.jobstores.base import JobStore
-from apscheduler.job import Job
-
-try:
-    from sqlalchemy import *
-except ImportError:  # pragma: nocover
-    raise ImportError('SQLAlchemyJobStore requires SQLAlchemy installed')
-
-logger = logging.getLogger(__name__)
-
-
-class SQLAlchemyJobStore(JobStore):
-    def __init__(self, url=None, engine=None, tablename='apscheduler_jobs',
-                 metadata=None, pickle_protocol=pickle.HIGHEST_PROTOCOL):
-        self.jobs = []
-        self.pickle_protocol = pickle_protocol
-
-        if engine:
-            self.engine = engine
-        elif url:
-            self.engine = create_engine(url)
-        else:
-            raise ValueError('Need either "engine" or "url" defined')
-
-        self.jobs_t = Table(tablename, metadata or MetaData(),
-            Column('id', Integer,
-                   Sequence(tablename + '_id_seq', optional=True),
-                   primary_key=True),
-            Column('trigger', PickleType(pickle_protocol, mutable=False),
-                   nullable=False),
-            Column('func_ref', String(1024), nullable=False),
-            Column('args', PickleType(pickle_protocol, mutable=False),
-                   nullable=False),
-            Column('kwargs', PickleType(pickle_protocol, mutable=False),
-                   nullable=False),
-            Column('name', Unicode(1024), unique=True),
-            Column('misfire_grace_time', Integer, nullable=False),
-            Column('coalesce', Boolean, nullable=False),
-            Column('max_runs', Integer),
-            Column('max_instances', Integer),
-            Column('next_run_time', DateTime, nullable=False),
-            Column('runs', BigInteger))
-
-        self.jobs_t.create(self.engine, True)
-
-    def add_job(self, job):
-        job_dict = job.__getstate__()
-        result = self.engine.execute(self.jobs_t.insert().values(**job_dict))
-        job.id = result.inserted_primary_key[0]
-        self.jobs.append(job)
-
-    def remove_job(self, job):
-        delete = self.jobs_t.delete().where(self.jobs_t.c.id == job.id)
-        self.engine.execute(delete)
-        self.jobs.remove(job)
-
-    def load_jobs(self):
-        jobs = []
-        for row in self.engine.execute(select([self.jobs_t])):
-            try:
-                job = Job.__new__(Job)
-                job_dict = dict(row.items())
-                job.__setstate__(job_dict)
-                jobs.append(job)
-            except Exception:
-                job_name = job_dict.get('name', '(unknown)')
-                logger.exception('Unable to restore job "%s"', job_name)
-        self.jobs = jobs
-
-    def update_job(self, job):
-        job_dict = job.__getstate__()
-        update = self.jobs_t.update().where(self.jobs_t.c.id == job.id).\
-            values(next_run_time=job_dict['next_run_time'],
-                   runs=job_dict['runs'])
-        self.engine.execute(update)
-
-    def close(self):
-        self.engine.dispose()
-
-    def __repr__(self):
-        return '<%s (url=%s)>' % (self.__class__.__name__, self.engine.url)

lib/apscheduler/scheduler.py

@@ -1,559 +0,0 @@
-"""
-This module is the main part of the library. It houses the Scheduler class
-and related exceptions.
-"""
-
-from threading import Thread, Event, Lock
-from datetime import datetime, timedelta
-from logging import getLogger
-import os
-import sys
-
-from apscheduler.util import *
-from apscheduler.triggers import SimpleTrigger, IntervalTrigger, CronTrigger
-from apscheduler.jobstores.ram_store import RAMJobStore
-from apscheduler.job import Job, MaxInstancesReachedError
-from apscheduler.events import *
-from apscheduler.threadpool import ThreadPool
-
-logger = getLogger(__name__)
-
-
-class SchedulerAlreadyRunningError(Exception):
-    """
-    Raised when attempting to start or configure the scheduler when it's
-    already running.
-    """
-
-    def __str__(self):
-        return 'Scheduler is already running'
-
-
-class Scheduler(object):
-    """
-    This class is responsible for scheduling jobs and triggering
-    their execution.
-    """
-
-    _stopped = False
-    _thread = None
-
-    def __init__(self, gconfig={}, **options):
-        self._wakeup = Event()
-        self._jobstores = {}
-        self._jobstores_lock = Lock()
-        self._listeners = []
-        self._listeners_lock = Lock()
-        self._pending_jobs = []
-        self.configure(gconfig, **options)
-
-    def configure(self, gconfig={}, **options):
-        """
-        Reconfigures the scheduler with the given options. Can only be done
-        when the scheduler isn't running.
-        """
-        if self.running:
-            raise SchedulerAlreadyRunningError
-
-        # Set general options
-        config = combine_opts(gconfig, 'apscheduler.', options)
-        self.misfire_grace_time = int(config.pop('misfire_grace_time', 1))
-        self.coalesce = asbool(config.pop('coalesce', True))
-        self.daemonic = asbool(config.pop('daemonic', True))
-
-        # Configure the thread pool
-        if 'threadpool' in config:
-            self._threadpool = maybe_ref(config['threadpool'])
-        else:
-            threadpool_opts = combine_opts(config, 'threadpool.')
-            self._threadpool = ThreadPool(**threadpool_opts)
-
-        # Configure job stores
-        jobstore_opts = combine_opts(config, 'jobstore.')
-        jobstores = {}
-        for key, value in jobstore_opts.items():
-            store_name, option = key.split('.', 1)
-            opts_dict = jobstores.setdefault(store_name, {})
-            opts_dict[option] = value
-
-        for alias, opts in jobstores.items():
-            classname = opts.pop('class')
-            cls = maybe_ref(classname)
-            jobstore = cls(**opts)
-            self.add_jobstore(jobstore, alias, True)
-
-    def start(self):
-        """
-        Starts the scheduler in a new thread.
-        """
-        if self.running:
-            raise SchedulerAlreadyRunningError
-
-        # Create a RAMJobStore as the default if there is no default job store
-        if not 'default' in self._jobstores:
-            self.add_jobstore(RAMJobStore(), 'default', True)
-
-        # Schedule all pending jobs
-        for job, jobstore in self._pending_jobs:
-            self._real_add_job(job, jobstore, False)
-        del self._pending_jobs[:]
-
-        self._stopped = False
-        self._thread = Thread(target=self._main_loop, name='APScheduler')
-        self._thread.setDaemon(self.daemonic)
-        self._thread.start()
-
-    def shutdown(self, wait=True, shutdown_threadpool=True):
-        """
-        Shuts down the scheduler and terminates the thread.
-        Does not interrupt any currently running jobs.
-
-        :param wait: ``True`` to wait until all currently executing jobs have
-                     finished (if ``shutdown_threadpool`` is also ``True``)
-        :param shutdown_threadpool: ``True`` to shut down the thread pool
-        """
-        if not self.running:
-            return
-
-        self._stopped = True
-        self._wakeup.set()
-
-        # Shut down the thread pool
-        if shutdown_threadpool:
-            self._threadpool.shutdown(wait)
-
-        # Wait until the scheduler thread terminates
-        self._thread.join()
-
-    @property
-    def running(self):
-        return not self._stopped and self._thread and self._thread.isAlive()
-
-    def add_jobstore(self, jobstore, alias, quiet=False):
-        """
-        Adds a job store to this scheduler.
-
-        :param jobstore: job store to be added
-        :param alias: alias for the job store
-        :param quiet: True to suppress scheduler thread wakeup
-        :type jobstore: instance of
-            :class:`~apscheduler.jobstores.base.JobStore`
-        :type alias: str
-        """
-        self._jobstores_lock.acquire()
-        try:
-            if alias in self._jobstores:
-                raise KeyError('Alias "%s" is already in use' % alias)
-            self._jobstores[alias] = jobstore
-            jobstore.load_jobs()
-        finally:
-            self._jobstores_lock.release()
-
-        # Notify listeners that a new job store has been added
-        self._notify_listeners(JobStoreEvent(EVENT_JOBSTORE_ADDED, alias))
-
-        # Notify the scheduler so it can scan the new job store for jobs
-        if not quiet:
-            self._wakeup.set()
-
-    def remove_jobstore(self, alias):
-        """
-        Removes the job store by the given alias from this scheduler.
-
-        :type alias: str
-        """
-        self._jobstores_lock.acquire()
-        try:
-            try:
-                del self._jobstores[alias]
-            except KeyError:
-                raise KeyError('No such job store: %s' % alias)
-        finally:
-            self._jobstores_lock.release()
-
-        # Notify listeners that a job store has been removed
-        self._notify_listeners(JobStoreEvent(EVENT_JOBSTORE_REMOVED, alias))
-
-    def add_listener(self, callback, mask=EVENT_ALL):
-        """
-        Adds a listener for scheduler events. When a matching event occurs,
-        ``callback`` is executed with the event object as its sole argument.
-        If the ``mask`` parameter is not provided, the callback will receive
-        events of all types.
-
-        :param callback: any callable that takes one argument
-        :param mask: bitmask that indicates which events should be listened to
-        """
-        self._listeners_lock.acquire()
-        try:
-            self._listeners.append((callback, mask))
-        finally:
-            self._listeners_lock.release()
-
-    def remove_listener(self, callback):
-        """
-        Removes a previously added event listener.
-        """
-        self._listeners_lock.acquire()
-        try:
-            for i, (cb, _) in enumerate(self._listeners):
-                if callback == cb:
-                    del self._listeners[i]
-        finally:
-            self._listeners_lock.release()
-
-    def _notify_listeners(self, event):
-        self._listeners_lock.acquire()
-        try:
-            listeners = tuple(self._listeners)
-        finally:
-            self._listeners_lock.release()
-
-        for cb, mask in listeners:
-            if event.code & mask:
-                try:
-                    cb(event)
-                except:
-                    logger.exception('Error notifying listener')
-
-    def _real_add_job(self, job, jobstore, wakeup):
-        job.compute_next_run_time(datetime.now())
-        if not job.next_run_time:
-            raise ValueError('Not adding job since it would never be run')
-
-        self._jobstores_lock.acquire()
-        try:
-            try:
-                store = self._jobstores[jobstore]
-            except KeyError:
-                raise KeyError('No such job store: %s' % jobstore)
-            store.add_job(job)
-        finally:
-            self._jobstores_lock.release()
-
-        # Notify listeners that a new job has been added
-        event = JobStoreEvent(EVENT_JOBSTORE_JOB_ADDED, jobstore, job)
-        self._notify_listeners(event)
-
-        logger.info('Added job "%s" to job store "%s"', job, jobstore)
-
-        # Notify the scheduler about the new job
-        if wakeup:
-            self._wakeup.set()
-
-    def add_job(self, trigger, func, args, kwargs, jobstore='default',
-                **options):
-        """
-        Adds the given job to the job list and notifies the scheduler thread.
-
-        :param trigger: alias of the job store to store the job in
-        :param func: callable to run at the given time
-        :param args: list of positional arguments to call func with
-        :param kwargs: dict of keyword arguments to call func with
-        :param jobstore: alias of the job store to store the job in
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        job = Job(trigger, func, args or [], kwargs or {},
-                  options.pop('misfire_grace_time', self.misfire_grace_time),
-                  options.pop('coalesce', self.coalesce), **options)
-        if not self.running:
-            self._pending_jobs.append((job, jobstore))
-            logger.info('Adding job tentatively -- it will be properly '
-                        'scheduled when the scheduler starts')
-        else:
-            self._real_add_job(job, jobstore, True)
-        return job
-
-    def _remove_job(self, job, alias, jobstore):
-        jobstore.remove_job(job)
-
-        # Notify listeners that a job has been removed
-        event = JobStoreEvent(EVENT_JOBSTORE_JOB_REMOVED, alias, job)
-        self._notify_listeners(event)
-
-        logger.info('Removed job "%s"', job)
-
-    def add_date_job(self, func, date, args=None, kwargs=None, **options):
-        """
-        Schedules a job to be completed on a specific date and time.
-
-        :param func: callable to run at the given time
-        :param date: the date/time to run the job at
-        :param name: name of the job
-        :param jobstore: stored the job in the named (or given) job store
-        :param misfire_grace_time: seconds after the designated run time that
-            the job is still allowed to be run
-        :type date: :class:`datetime.date`
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        trigger = SimpleTrigger(date)
-        return self.add_job(trigger, func, args, kwargs, **options)
-
-    def add_interval_job(self, func, weeks=0, days=0, hours=0, minutes=0,
-                         seconds=0, start_date=None, args=None, kwargs=None,
-                         **options):
-        """
-        Schedules a job to be completed on specified intervals.
-
-        :param func: callable to run
-        :param weeks: number of weeks to wait
-        :param days: number of days to wait
-        :param hours: number of hours to wait
-        :param minutes: number of minutes to wait
-        :param seconds: number of seconds to wait
-        :param start_date: when to first execute the job and start the
-            counter (default is after the given interval)
-        :param args: list of positional arguments to call func with
-        :param kwargs: dict of keyword arguments to call func with
-        :param name: name of the job
-        :param jobstore: alias of the job store to add the job to
-        :param misfire_grace_time: seconds after the designated run time that
-            the job is still allowed to be run
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        interval = timedelta(weeks=weeks, days=days, hours=hours,
-                             minutes=minutes, seconds=seconds)
-        trigger = IntervalTrigger(interval, start_date)
-        return self.add_job(trigger, func, args, kwargs, **options)
-
-    def add_cron_job(self, func, year='*', month='*', day='*', week='*',
-                     day_of_week='*', hour='*', minute='*', second='*',
-                     start_date=None, args=None, kwargs=None, **options):
-        """
-        Schedules a job to be completed on times that match the given
-        expressions.
-
-        :param func: callable to run
-        :param year: year to run on
-        :param month: month to run on (0 = January)
-        :param day: day of month to run on
-        :param week: week of the year to run on
-        :param day_of_week: weekday to run on (0 = Monday)
-        :param hour: hour to run on
-        :param second: second to run on
-        :param args: list of positional arguments to call func with
-        :param kwargs: dict of keyword arguments to call func with
-        :param name: name of the job
-        :param jobstore: alias of the job store to add the job to
-        :param misfire_grace_time: seconds after the designated run time that
-            the job is still allowed to be run
-        :return: the scheduled job
-        :rtype: :class:`~apscheduler.job.Job`
-        """
-        trigger = CronTrigger(year=year, month=month, day=day, week=week,
-                              day_of_week=day_of_week, hour=hour,
-                              minute=minute, second=second,
-                              start_date=start_date)
-        return self.add_job(trigger, func, args, kwargs, **options)
-
-    def cron_schedule(self, **options):
-        """
-        Decorator version of :meth:`add_cron_job`.
-        This decorator does not wrap its host function.
-        Unscheduling decorated functions is possible by passing the ``job``
-        attribute of the scheduled function to :meth:`unschedule_job`.
-        """
-        def inner(func):
-            func.job = self.add_cron_job(func, **options)
-            return func
-        return inner
-
-    def interval_schedule(self, **options):
-        """
-        Decorator version of :meth:`add_interval_job`.
-        This decorator does not wrap its host function.
-        Unscheduling decorated functions is possible by passing the ``job``
-        attribute of the scheduled function to :meth:`unschedule_job`.
-        """
-        def inner(func):
-            func.job = self.add_interval_job(func, **options)
-            return func
-        return inner
-
-    def get_jobs(self):
-        """
-        Returns a list of all scheduled jobs.
-
-        :return: list of :class:`~apscheduler.job.Job` objects
-        """
-        self._jobstores_lock.acquire()
-        try:
-            jobs = []
-            for jobstore in itervalues(self._jobstores):
-                jobs.extend(jobstore.jobs)
-            return jobs
-        finally:
-            self._jobstores_lock.release()
-
-    def unschedule_job(self, job):
-        """
-        Removes a job, preventing it from being run any more.
-        """
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                if job in list(jobstore.jobs):
-                    self._remove_job(job, alias, jobstore)
-                    return
-        finally:
-            self._jobstores_lock.release()
-
-        raise KeyError('Job "%s" is not scheduled in any job store' % job)
-
-    def unschedule_func(self, func):
-        """
-        Removes all jobs that would execute the given function.
-        """
-        found = False
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                for job in list(jobstore.jobs):
-                    if job.func == func:
-                        self._remove_job(job, alias, jobstore)
-                        found = True
-        finally:
-            self._jobstores_lock.release()
-
-        if not found:
-            raise KeyError('The given function is not scheduled in this '
-                           'scheduler')
-
-    def print_jobs(self, out=None):
-        """
-        Prints out a textual listing of all jobs currently scheduled on this
-        scheduler.
-
-        :param out: a file-like object to print to (defaults to **sys.stdout**
-                    if nothing is given)
-        """
-        out = out or sys.stdout
-        job_strs = []
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                job_strs.append('Jobstore %s:' % alias)
-                if jobstore.jobs:
-                    for job in jobstore.jobs:
-                        job_strs.append('    %s' % job)
-                else:
-                    job_strs.append('    No scheduled jobs')
-        finally:
-            self._jobstores_lock.release()
-
-        out.write(os.linesep.join(job_strs))
-
-    def _run_job(self, job, run_times):
-        """
-        Acts as a harness that runs the actual job code in a thread.
-        """
-        for run_time in run_times:
-            # See if the job missed its run time window, and handle possible
-            # misfires accordingly
-            difference = datetime.now() - run_time
-            grace_time = timedelta(seconds=job.misfire_grace_time)
-            if difference > grace_time:
-                # Notify listeners about a missed run
-                event = JobEvent(EVENT_JOB_MISSED, job, run_time)
-                self._notify_listeners(event)
-                logger.warning('Run time of job "%s" was missed by %s',
-                               job, difference)
-            else:
-                try:
-                    job.add_instance()
-                except MaxInstancesReachedError:
-                    event = JobEvent(EVENT_JOB_MISSED, job, run_time)
-                    self._notify_listeners(event)
-                    logger.warning('Execution of job "%s" skipped: '
-                                   'maximum number of running instances '
-                                   'reached (%d)', job, job.max_instances)
-                    break
-
-                logger.info('Running job "%s" (scheduled at %s)', job,
-                            run_time)
-
-                try:
-                    retval = job.func(*job.args, **job.kwargs)
-                except:
-                    # Notify listeners about the exception
-                    exc, tb = sys.exc_info()[1:]
-                    event = JobEvent(EVENT_JOB_ERROR, job, run_time,
-                                     exception=exc, traceback=tb)
-                    self._notify_listeners(event)
-
-                    logger.exception('Job "%s" raised an exception', job)
-                else:
-                    # Notify listeners about successful execution
-                    event = JobEvent(EVENT_JOB_EXECUTED, job, run_time,
-                                     retval=retval)
-                    self._notify_listeners(event)
-
-                    logger.info('Job "%s" executed successfully', job)
-
-                job.remove_instance()
-
-                # If coalescing is enabled, don't attempt any further runs
-                if job.coalesce:
-                    break
-
-    def _process_jobs(self, now):
-        """
-        Iterates through jobs in every jobstore, starts pending jobs
-        and figures out the next wakeup time.
-        """
-        next_wakeup_time = None
-        self._jobstores_lock.acquire()
-        try:
-            for alias, jobstore in iteritems(self._jobstores):
-                for job in tuple(jobstore.jobs):
-                    run_times = job.get_run_times(now)
-                    if run_times:
-                        self._threadpool.submit(self._run_job, job, run_times)
-
-                        # Increase the job's run count
-                        if job.coalesce:
-                            job.runs += 1
-                        else:
-                            job.runs += len(run_times)
-
-                        # Update the job, but don't keep finished jobs around
-                        if job.compute_next_run_time(now + timedelta(microseconds=1)):
-                            jobstore.update_job(job)
-                        else:
-                            self._remove_job(job, alias, jobstore)
-
-                    if not next_wakeup_time:
-                        next_wakeup_time = job.next_run_time
-                    elif job.next_run_time:
-                        next_wakeup_time = min(next_wakeup_time,
-                                               job.next_run_time)
-            return next_wakeup_time
-        finally:
-            self._jobstores_lock.release()
-
-    def _main_loop(self):
-        """Executes jobs on schedule."""
-
-        logger.info('Scheduler started')
-        self._notify_listeners(SchedulerEvent(EVENT_SCHEDULER_START))
-
-        self._wakeup.clear()
-        while not self._stopped:
-            logger.debug('Looking for jobs to run')
-            now = datetime.now()
-            next_wakeup_time = self._process_jobs(now)
-
-            # Sleep until the next job is scheduled to be run,
-            # a new job is added or the scheduler is stopped
-            if next_wakeup_time is not None:
-                wait_seconds = time_difference(next_wakeup_time, now)
-                logger.debug('Next wakeup is due at %s (in %f seconds)',
-                             next_wakeup_time, wait_seconds)
-                self._wakeup.wait(wait_seconds)
-            else:
-                logger.debug('No jobs; waiting until a job is added')
-                self._wakeup.wait()
-            self._wakeup.clear()
-
-        logger.info('Scheduler has been shut down')
-        self._notify_listeners(SchedulerEvent(EVENT_SCHEDULER_SHUTDOWN))

lib/apscheduler/schedulers/__init__.py

@@ -0,0 +1,12 @@
+class SchedulerAlreadyRunningError(Exception):
+    """Raised when attempting to start or configure the scheduler when it's already running."""
+
+    def __str__(self):
+        return 'Scheduler is already running'
+
+
+class SchedulerNotRunningError(Exception):
+    """Raised when attempting to shutdown the scheduler when it's not running."""
+
+    def __str__(self):
+        return 'Scheduler is not running'

lib/apscheduler/schedulers/asyncio.py

@@ -0,0 +1,68 @@
+from __future__ import absolute_import
+from functools import wraps
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.util import maybe_ref
+
+try:
+    import asyncio
+except ImportError:  # pragma: nocover
+    try:
+        import trollius as asyncio
+    except ImportError:
+        raise ImportError('AsyncIOScheduler requires either Python 3.4 or the asyncio package installed')
+
+
+def run_in_event_loop(func):
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        self._eventloop.call_soon_threadsafe(func, self, *args, **kwargs)
+    return wrapper
+
+
+class AsyncIOScheduler(BaseScheduler):
+    """
+    A scheduler that runs on an asyncio (:pep:`3156`) event loop.
+
+    Extra options:
+
+    ============== =============================================================
+    ``event_loop`` AsyncIO event loop to use (defaults to the global event loop)
+    ============== =============================================================
+    """
+
+    _eventloop = None
+    _timeout = None
+
+    def start(self):
+        super(AsyncIOScheduler, self).start()
+        self.wakeup()
+
+    @run_in_event_loop
+    def shutdown(self, wait=True):
+        super(AsyncIOScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _configure(self, config):
+        self._eventloop = maybe_ref(config.pop('event_loop', None)) or asyncio.get_event_loop()
+        super(AsyncIOScheduler, self)._configure(config)
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._timeout = self._eventloop.call_later(wait_seconds, self.wakeup)
+
+    def _stop_timer(self):
+        if self._timeout:
+            self._timeout.cancel()
+            del self._timeout
+
+    @run_in_event_loop
+    def wakeup(self):
+        self._stop_timer()
+        wait_seconds = self._process_jobs()
+        self._start_timer(wait_seconds)
+
+    def _create_default_executor(self):
+        from apscheduler.executors.asyncio import AsyncIOExecutor
+        return AsyncIOExecutor()

lib/apscheduler/schedulers/background.py

@@ -0,0 +1,39 @@
+from __future__ import absolute_import
+from threading import Thread, Event
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.schedulers.blocking import BlockingScheduler
+from apscheduler.util import asbool
+
+
+class BackgroundScheduler(BlockingScheduler):
+    """
+    A scheduler that runs in the background using a separate thread
+    (:meth:`~apscheduler.schedulers.base.BaseScheduler.start` will return immediately).
+
+    Extra options:
+
+    ========== ============================================================================================
+    ``daemon`` Set the ``daemon`` option in the background thread (defaults to ``True``,
+               see `the documentation <https://docs.python.org/3.4/library/threading.html#thread-objects>`_
+               for further details)
+    ========== ============================================================================================
+    """
+
+    _thread = None
+
+    def _configure(self, config):
+        self._daemon = asbool(config.pop('daemon', True))
+        super(BackgroundScheduler, self)._configure(config)
+
+    def start(self):
+        BaseScheduler.start(self)
+        self._event = Event()
+        self._thread = Thread(target=self._main_loop, name='APScheduler')
+        self._thread.daemon = self._daemon
+        self._thread.start()
+
+    def shutdown(self, wait=True):
+        super(BackgroundScheduler, self).shutdown(wait)
+        self._thread.join()
+        del self._thread

lib/apscheduler/schedulers/base.py

@@ -0,0 +1,845 @@
+from __future__ import print_function
+from abc import ABCMeta, abstractmethod
+from collections import MutableMapping
+from threading import RLock
+from datetime import datetime
+from logging import getLogger
+import sys
+
+from pkg_resources import iter_entry_points
+from tzlocal import get_localzone
+import six
+
+from apscheduler.schedulers import SchedulerAlreadyRunningError, SchedulerNotRunningError
+from apscheduler.executors.base import MaxInstancesReachedError, BaseExecutor
+from apscheduler.executors.pool import ThreadPoolExecutor
+from apscheduler.jobstores.base import ConflictingIdError, JobLookupError, BaseJobStore
+from apscheduler.jobstores.memory import MemoryJobStore
+from apscheduler.job import Job
+from apscheduler.triggers.base import BaseTrigger
+from apscheduler.util import asbool, asint, astimezone, maybe_ref, timedelta_seconds, undefined
+from apscheduler.events import (
+    SchedulerEvent, JobEvent, EVENT_SCHEDULER_START, EVENT_SCHEDULER_SHUTDOWN, EVENT_JOBSTORE_ADDED,
+    EVENT_JOBSTORE_REMOVED, EVENT_ALL, EVENT_JOB_MODIFIED, EVENT_JOB_REMOVED, EVENT_JOB_ADDED, EVENT_EXECUTOR_ADDED,
+    EVENT_EXECUTOR_REMOVED, EVENT_ALL_JOBS_REMOVED)
+
+
+class BaseScheduler(six.with_metaclass(ABCMeta)):
+    """
+    Abstract base class for all schedulers. Takes the following keyword arguments:
+
+    :param str|logging.Logger logger: logger to use for the scheduler's logging (defaults to apscheduler.scheduler)
+    :param str|datetime.tzinfo timezone: the default time zone (defaults to the local timezone)
+    :param dict job_defaults: default values for newly added jobs
+    :param dict jobstores: a dictionary of job store alias -> job store instance or configuration dict
+    :param dict executors: a dictionary of executor alias -> executor instance or configuration dict
+
+    .. seealso:: :ref:`scheduler-config`
+    """
+
+    _trigger_plugins = dict((ep.name, ep) for ep in iter_entry_points('apscheduler.triggers'))
+    _trigger_classes = {}
+    _executor_plugins = dict((ep.name, ep) for ep in iter_entry_points('apscheduler.executors'))
+    _executor_classes = {}
+    _jobstore_plugins = dict((ep.name, ep) for ep in iter_entry_points('apscheduler.jobstores'))
+    _jobstore_classes = {}
+    _stopped = True
+
+    #
+    # Public API
+    #
+
+    def __init__(self, gconfig={}, **options):
+        super(BaseScheduler, self).__init__()
+        self._executors = {}
+        self._executors_lock = self._create_lock()
+        self._jobstores = {}
+        self._jobstores_lock = self._create_lock()
+        self._listeners = []
+        self._listeners_lock = self._create_lock()
+        self._pending_jobs = []
+        self.configure(gconfig, **options)
+
+    def configure(self, gconfig={}, prefix='apscheduler.', **options):
+        """
+        Reconfigures the scheduler with the given options. Can only be done when the scheduler isn't running.
+
+        :param dict gconfig: a "global" configuration dictionary whose values can be overridden by keyword arguments to
+                             this method
+        :param str|unicode prefix: pick only those keys from ``gconfig`` that are prefixed with this string
+                                   (pass an empty string or ``None`` to use all keys)
+        :raises SchedulerAlreadyRunningError: if the scheduler is already running
+        """
+
+        if self.running:
+            raise SchedulerAlreadyRunningError
+
+        # If a non-empty prefix was given, strip it from the keys in the global configuration dict
+        if prefix:
+            prefixlen = len(prefix)
+            gconfig = dict((key[prefixlen:], value) for key, value in six.iteritems(gconfig) if key.startswith(prefix))
+
+        # Create a structure from the dotted options (e.g. "a.b.c = d" -> {'a': {'b': {'c': 'd'}}})
+        config = {}
+        for key, value in six.iteritems(gconfig):
+            parts = key.split('.')
+            parent = config
+            key = parts.pop(0)
+            while parts:
+                parent = parent.setdefault(key, {})
+                key = parts.pop(0)
+            parent[key] = value
+
+        # Override any options with explicit keyword arguments
+        config.update(options)
+        self._configure(config)
+
+    @abstractmethod
+    def start(self):
+        """
+        Starts the scheduler. The details of this process depend on the implementation.
+
+        :raises SchedulerAlreadyRunningError: if the scheduler is already running
+        """
+
+        if self.running:
+            raise SchedulerAlreadyRunningError
+
+        with self._executors_lock:
+            # Create a default executor if nothing else is configured
+            if 'default' not in self._executors:
+                self.add_executor(self._create_default_executor(), 'default')
+
+            # Start all the executors
+            for alias, executor in six.iteritems(self._executors):
+                executor.start(self, alias)
+
+        with self._jobstores_lock:
+            # Create a default job store if nothing else is configured
+            if 'default' not in self._jobstores:
+                self.add_jobstore(self._create_default_jobstore(), 'default')
+
+            # Start all the job stores
+            for alias, store in six.iteritems(self._jobstores):
+                store.start(self, alias)
+
+            # Schedule all pending jobs
+            for job, jobstore_alias, replace_existing in self._pending_jobs:
+                self._real_add_job(job, jobstore_alias, replace_existing, False)
+            del self._pending_jobs[:]
+
+        self._stopped = False
+        self._logger.info('Scheduler started')
+
+        # Notify listeners that the scheduler has been started
+        self._dispatch_event(SchedulerEvent(EVENT_SCHEDULER_START))
+
+    @abstractmethod
+    def shutdown(self, wait=True):
+        """
+        Shuts down the scheduler. Does not interrupt any currently running jobs.
+
+        :param bool wait: ``True`` to wait until all currently executing jobs have finished
+        :raises SchedulerNotRunningError: if the scheduler has not been started yet
+        """
+
+        if not self.running:
+            raise SchedulerNotRunningError
+
+        self._stopped = True
+
+        # Shut down all executors
+        for executor in six.itervalues(self._executors):
+            executor.shutdown(wait)
+
+        # Shut down all job stores
+        for jobstore in six.itervalues(self._jobstores):
+            jobstore.shutdown()
+
+        self._logger.info('Scheduler has been shut down')
+        self._dispatch_event(SchedulerEvent(EVENT_SCHEDULER_SHUTDOWN))
+
+    @property
+    def running(self):
+        return not self._stopped
+
+    def add_executor(self, executor, alias='default', **executor_opts):
+        """
+        Adds an executor to this scheduler. Any extra keyword arguments will be passed to the executor plugin's
+        constructor, assuming that the first argument is the name of an executor plugin.
+
+        :param str|unicode|apscheduler.executors.base.BaseExecutor executor: either an executor instance or the name of
+            an executor plugin
+        :param str|unicode alias: alias for the scheduler
+        :raises ValueError: if there is already an executor by the given alias
+        """
+
+        with self._executors_lock:
+            if alias in self._executors:
+                raise ValueError('This scheduler already has an executor by the alias of "%s"' % alias)
+
+            if isinstance(executor, BaseExecutor):
+                self._executors[alias] = executor
+            elif isinstance(executor, six.string_types):
+                self._executors[alias] = executor = self._create_plugin_instance('executor', executor, executor_opts)
+            else:
+                raise TypeError('Expected an executor instance or a string, got %s instead' %
+                                executor.__class__.__name__)
+
+            # Start the executor right away if the scheduler is running
+            if self.running:
+                executor.start(self)
+
+        self._dispatch_event(SchedulerEvent(EVENT_EXECUTOR_ADDED, alias))
+
+    def remove_executor(self, alias, shutdown=True):
+        """
+        Removes the executor by the given alias from this scheduler.
+
+        :param str|unicode alias: alias of the executor
+        :param bool shutdown: ``True`` to shut down the executor after removing it
+        """
+
+        with self._jobstores_lock:
+            executor = self._lookup_executor(alias)
+            del self._executors[alias]
+
+        if shutdown:
+            executor.shutdown()
+
+        self._dispatch_event(SchedulerEvent(EVENT_EXECUTOR_REMOVED, alias))
+
+    def add_jobstore(self, jobstore, alias='default', **jobstore_opts):
+        """
+        Adds a job store to this scheduler. Any extra keyword arguments will be passed to the job store plugin's
+        constructor, assuming that the first argument is the name of a job store plugin.
+
+        :param str|unicode|apscheduler.jobstores.base.BaseJobStore jobstore: job store to be added
+        :param str|unicode alias: alias for the job store
+        :raises ValueError: if there is already a job store by the given alias
+        """
+
+        with self._jobstores_lock:
+            if alias in self._jobstores:
+                raise ValueError('This scheduler already has a job store by the alias of "%s"' % alias)
+
+            if isinstance(jobstore, BaseJobStore):
+                self._jobstores[alias] = jobstore
+            elif isinstance(jobstore, six.string_types):
+                self._jobstores[alias] = jobstore = self._create_plugin_instance('jobstore', jobstore, jobstore_opts)
+            else:
+                raise TypeError('Expected a job store instance or a string, got %s instead' %
+                                jobstore.__class__.__name__)
+
+            # Start the job store right away if the scheduler is running
+            if self.running:
+                jobstore.start(self, alias)
+
+        # Notify listeners that a new job store has been added
+        self._dispatch_event(SchedulerEvent(EVENT_JOBSTORE_ADDED, alias))
+
+        # Notify the scheduler so it can scan the new job store for jobs
+        if self.running:
+            self.wakeup()
+
+    def remove_jobstore(self, alias, shutdown=True):
+        """
+        Removes the job store by the given alias from this scheduler.
+
+        :param str|unicode alias: alias of the job store
+        :param bool shutdown: ``True`` to shut down the job store after removing it
+        """
+
+        with self._jobstores_lock:
+            jobstore = self._lookup_jobstore(alias)
+            del self._jobstores[alias]
+
+        if shutdown:
+            jobstore.shutdown()
+
+        self._dispatch_event(SchedulerEvent(EVENT_JOBSTORE_REMOVED, alias))
+
+    def add_listener(self, callback, mask=EVENT_ALL):
+        """
+        add_listener(callback, mask=EVENT_ALL)
+
+        Adds a listener for scheduler events. When a matching event occurs, ``callback`` is executed with the event
+        object as its sole argument. If the ``mask`` parameter is not provided, the callback will receive events of all
+        types.
+
+        :param callback: any callable that takes one argument
+        :param int mask: bitmask that indicates which events should be listened to
+
+        .. seealso:: :mod:`apscheduler.events`
+        .. seealso:: :ref:`scheduler-events`
+        """
+
+        with self._listeners_lock:
+            self._listeners.append((callback, mask))
+
+    def remove_listener(self, callback):
+        """Removes a previously added event listener."""
+
+        with self._listeners_lock:
+            for i, (cb, _) in enumerate(self._listeners):
+                if callback == cb:
+                    del self._listeners[i]
+
+    def add_job(self, func, trigger=None, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined,
+                coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default',
+                executor='default', replace_existing=False, **trigger_args):
+        """
+        add_job(func, trigger=None, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined, \
+            coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default', \
+            executor='default', replace_existing=False, **trigger_args)
+
+        Adds the given job to the job list and wakes up the scheduler if it's already running.
+
+        Any option that defaults to ``undefined`` will be replaced with the corresponding default value when the job is
+        scheduled (which happens when the scheduler is started, or immediately if the scheduler is already running).
+
+        The ``func`` argument can be given either as a callable object or a textual reference in the
+        ``package.module:some.object`` format, where the first half (separated by ``:``) is an importable module and the
+        second half is a reference to the callable object, relative to the module.
+
+        The ``trigger`` argument can either be:
+          #. the alias name of the trigger (e.g. ``date``, ``interval`` or ``cron``), in which case any extra keyword
+             arguments to this method are passed on to the trigger's constructor
+          #. an instance of a trigger class
+
+        :param func: callable (or a textual reference to one) to run at the given time
+        :param str|apscheduler.triggers.base.BaseTrigger trigger: trigger that determines when ``func`` is called
+        :param list|tuple args: list of positional arguments to call func with
+        :param dict kwargs: dict of keyword arguments to call func with
+        :param str|unicode id: explicit identifier for the job (for modifying it later)
+        :param str|unicode name: textual description of the job
+        :param int misfire_grace_time: seconds after the designated run time that the job is still allowed to be run
+        :param bool coalesce: run once instead of many times if the scheduler determines that the job should be run more
+                              than once in succession
+        :param int max_instances: maximum number of concurrently running instances allowed for this job
+        :param datetime next_run_time: when to first run the job, regardless of the trigger (pass ``None`` to add the
+                                       job as paused)
+        :param str|unicode jobstore: alias of the job store to store the job in
+        :param str|unicode executor: alias of the executor to run the job with
+        :param bool replace_existing: ``True`` to replace an existing job with the same ``id`` (but retain the
+                                      number of runs from the existing one)
+        :rtype: Job
+        """
+
+        job_kwargs = {
+            'trigger': self._create_trigger(trigger, trigger_args),
+            'executor': executor,
+            'func': func,
+            'args': tuple(args) if args is not None else (),
+            'kwargs': dict(kwargs) if kwargs is not None else {},
+            'id': id,
+            'name': name,
+            'misfire_grace_time': misfire_grace_time,
+            'coalesce': coalesce,
+            'max_instances': max_instances,
+            'next_run_time': next_run_time
+        }
+        job_kwargs = dict((key, value) for key, value in six.iteritems(job_kwargs) if value is not undefined)
+        job = Job(self, **job_kwargs)
+
+        # Don't really add jobs to job stores before the scheduler is up and running
+        with self._jobstores_lock:
+            if not self.running:
+                self._pending_jobs.append((job, jobstore, replace_existing))
+                self._logger.info('Adding job tentatively -- it will be properly scheduled when the scheduler starts')
+            else:
+                self._real_add_job(job, jobstore, replace_existing, True)
+
+        return job
+
+    def scheduled_job(self, trigger, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined,
+                      coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default',
+                      executor='default', **trigger_args):
+        """
+        scheduled_job(trigger, args=None, kwargs=None, id=None, name=None, misfire_grace_time=undefined, \
+            coalesce=undefined, max_instances=undefined, next_run_time=undefined, jobstore='default', \
+            executor='default',**trigger_args)
+
+        A decorator version of :meth:`add_job`, except that ``replace_existing`` is always ``True``.
+
+        .. important:: The ``id`` argument must be given if scheduling a job in a persistent job store. The scheduler
+           cannot, however, enforce this requirement.
+        """
+
+        def inner(func):
+            self.add_job(func, trigger, args, kwargs, id, name, misfire_grace_time, coalesce, max_instances,
+                         next_run_time, jobstore, executor, True, **trigger_args)
+            return func
+        return inner
+
+    def modify_job(self, job_id, jobstore=None, **changes):
+        """
+        Modifies the properties of a single job. Modifications are passed to this method as extra keyword arguments.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        """
+        with self._jobstores_lock:
+            job, jobstore = self._lookup_job(job_id, jobstore)
+            job._modify(**changes)
+            if jobstore:
+                self._lookup_jobstore(jobstore).update_job(job)
+
+        self._dispatch_event(JobEvent(EVENT_JOB_MODIFIED, job_id, jobstore))
+
+        # Wake up the scheduler since the job's next run time may have been changed
+        self.wakeup()
+
+    def reschedule_job(self, job_id, jobstore=None, trigger=None, **trigger_args):
+        """
+        Constructs a new trigger for a job and updates its next run time.
+        Extra keyword arguments are passed directly to the trigger's constructor.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        :param trigger: alias of the trigger type or a trigger instance
+        """
+
+        trigger = self._create_trigger(trigger, trigger_args)
+        now = datetime.now(self.timezone)
+        next_run_time = trigger.get_next_fire_time(None, now)
+        self.modify_job(job_id, jobstore, trigger=trigger, next_run_time=next_run_time)
+
+    def pause_job(self, job_id, jobstore=None):
+        """
+        Causes the given job not to be executed until it is explicitly resumed.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        """
+
+        self.modify_job(job_id, jobstore, next_run_time=None)
+
+    def resume_job(self, job_id, jobstore=None):
+        """
+        Resumes the schedule of the given job, or removes the job if its schedule is finished.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        """
+
+        with self._jobstores_lock:
+            job, jobstore = self._lookup_job(job_id, jobstore)
+            now = datetime.now(self.timezone)
+            next_run_time = job.trigger.get_next_fire_time(None, now)
+            if next_run_time:
+                self.modify_job(job_id, jobstore, next_run_time=next_run_time)
+            else:
+                self.remove_job(job.id, jobstore)
+
+    def get_jobs(self, jobstore=None, pending=None):
+        """
+        Returns a list of pending jobs (if the scheduler hasn't been started yet) and scheduled jobs, either from a
+        specific job store or from all of them.
+
+        :param str|unicode jobstore: alias of the job store
+        :param bool pending: ``False`` to leave out pending jobs (jobs that are waiting for the scheduler start to be
+                             added to their respective job stores), ``True`` to only include pending jobs, anything else
+                             to return both
+        :rtype: list[Job]
+        """
+
+        with self._jobstores_lock:
+            jobs = []
+
+            if pending is not False:
+                for job, alias, replace_existing in self._pending_jobs:
+                    if jobstore is None or alias == jobstore:
+                        jobs.append(job)
+
+            if pending is not True:
+                for alias, store in six.iteritems(self._jobstores):
+                    if jobstore is None or alias == jobstore:
+                        jobs.extend(store.get_all_jobs())
+
+            return jobs
+
+    def get_job(self, job_id, jobstore=None):
+        """
+        Returns the Job that matches the given ``job_id``.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that most likely contains the job
+        :return: the Job by the given ID, or ``None`` if it wasn't found
+        :rtype: Job
+        """
+
+        with self._jobstores_lock:
+            try:
+                return self._lookup_job(job_id, jobstore)[0]
+            except JobLookupError:
+                return
+
+    def remove_job(self, job_id, jobstore=None):
+        """
+        Removes a job, preventing it from being run any more.
+
+        :param str|unicode job_id: the identifier of the job
+        :param str|unicode jobstore: alias of the job store that contains the job
+        :raises JobLookupError: if the job was not found
+        """
+
+        with self._jobstores_lock:
+            # Check if the job is among the pending jobs
+            for i, (job, jobstore_alias, replace_existing) in enumerate(self._pending_jobs):
+                if job.id == job_id:
+                    del self._pending_jobs[i]
+                    jobstore = jobstore_alias
+                    break
+            else:
+                # Otherwise, try to remove it from each store until it succeeds or we run out of stores to check
+                for alias, store in six.iteritems(self._jobstores):
+                    if jobstore in (None, alias):
+                        try:
+                            store.remove_job(job_id)
+                        except JobLookupError:
+                            continue
+
+                        jobstore = alias
+                        break
+
+        if jobstore is None:
+            raise JobLookupError(job_id)
+
+        # Notify listeners that a job has been removed
+        event = JobEvent(EVENT_JOB_REMOVED, job_id, jobstore)
+        self._dispatch_event(event)
+
+        self._logger.info('Removed job %s', job_id)
+
+    def remove_all_jobs(self, jobstore=None):
+        """
+        Removes all jobs from the specified job store, or all job stores if none is given.
+
+        :param str|unicode jobstore: alias of the job store
+        """
+
+        with self._jobstores_lock:
+            if jobstore:
+                self._pending_jobs = [pending for pending in self._pending_jobs if pending[1] != jobstore]
+            else:
+                self._pending_jobs = []
+
+            for alias, store in six.iteritems(self._jobstores):
+                if jobstore in (None, alias):
+                    store.remove_all_jobs()
+
+        self._dispatch_event(SchedulerEvent(EVENT_ALL_JOBS_REMOVED, jobstore))
+
+    def print_jobs(self, jobstore=None, out=None):
+        """
+        print_jobs(jobstore=None, out=sys.stdout)
+
+        Prints out a textual listing of all jobs currently scheduled on either all job stores or just a specific one.
+
+        :param str|unicode jobstore: alias of the job store, ``None`` to list jobs from all stores
+        :param file out: a file-like object to print to (defaults to **sys.stdout** if nothing is given)
+        """
+
+        out = out or sys.stdout
+        with self._jobstores_lock:
+            if self._pending_jobs:
+                print(six.u('Pending jobs:'), file=out)
+                for job, jobstore_alias, replace_existing in self._pending_jobs:
+                    if jobstore in (None, jobstore_alias):
+                        print(six.u('    %s') % job, file=out)
+
+            for alias, store in six.iteritems(self._jobstores):
+                if jobstore in (None, alias):
+                    print(six.u('Jobstore %s:') % alias, file=out)
+                    jobs = store.get_all_jobs()
+                    if jobs:
+                        for job in jobs:
+                            print(six.u('    %s') % job, file=out)
+                    else:
+                        print(six.u('    No scheduled jobs'), file=out)
+
+    @abstractmethod
+    def wakeup(self):
+        """
+        Notifies the scheduler that there may be jobs due for execution.
+        Triggers :meth:`_process_jobs` to be run in an implementation specific manner.
+        """
+
+    #
+    # Private API
+    #
+
+    def _configure(self, config):
+        # Set general options
+        self._logger = maybe_ref(config.pop('logger', None)) or getLogger('apscheduler.scheduler')
+        self.timezone = astimezone(config.pop('timezone', None)) or get_localzone()
+
+        # Set the job defaults
+        job_defaults = config.get('job_defaults', {})
+        self._job_defaults = {
+            'misfire_grace_time': asint(job_defaults.get('misfire_grace_time', 1)),
+            'coalesce': asbool(job_defaults.get('coalesce', True)),
+            'max_instances': asint(job_defaults.get('max_instances', 1))
+        }
+
+        # Configure executors
+        self._executors.clear()
+        for alias, value in six.iteritems(config.get('executors', {})):
+            if isinstance(value, BaseExecutor):
+                self.add_executor(value, alias)
+            elif isinstance(value, MutableMapping):
+                executor_class = value.pop('class', None)
+                plugin = value.pop('type', None)
+                if plugin:
+                    executor = self._create_plugin_instance('executor', plugin, value)
+                elif executor_class:
+                    cls = maybe_ref(executor_class)
+                    executor = cls(**value)
+                else:
+                    raise ValueError('Cannot create executor "%s" -- either "type" or "class" must be defined' % alias)
+
+                self.add_executor(executor, alias)
+            else:
+                raise TypeError("Expected executor instance or dict for executors['%s'], got %s instead" % (
+                    alias, value.__class__.__name__))
+
+        # Configure job stores
+        self._jobstores.clear()
+        for alias, value in six.iteritems(config.get('jobstores', {})):
+            if isinstance(value, BaseJobStore):
+                self.add_jobstore(value, alias)
+            elif isinstance(value, MutableMapping):
+                jobstore_class = value.pop('class', None)
+                plugin = value.pop('type', None)
+                if plugin:
+                    jobstore = self._create_plugin_instance('jobstore', plugin, value)
+                elif jobstore_class:
+                    cls = maybe_ref(jobstore_class)
+                    jobstore = cls(**value)
+                else:
+                    raise ValueError('Cannot create job store "%s" -- either "type" or "class" must be defined' % alias)
+
+                self.add_jobstore(jobstore, alias)
+            else:
+                raise TypeError("Expected job store instance or dict for jobstores['%s'], got %s instead" % (
+                    alias, value.__class__.__name__))
+
+    def _create_default_executor(self):
+        """Creates a default executor store, specific to the particular scheduler type."""
+
+        return ThreadPoolExecutor()
+
+    def _create_default_jobstore(self):
+        """Creates a default job store, specific to the particular scheduler type."""
+
+        return MemoryJobStore()
+
+    def _lookup_executor(self, alias):
+        """
+        Returns the executor instance by the given name from the list of executors that were added to this scheduler.
+
+        :type alias: str
+        :raises KeyError: if no executor by the given alias is not found
+        """
+
+        try:
+            return self._executors[alias]
+        except KeyError:
+            raise KeyError('No such executor: %s' % alias)
+
+    def _lookup_jobstore(self, alias):
+        """
+        Returns the job store instance by the given name from the list of job stores that were added to this scheduler.
+
+        :type alias: str
+        :raises KeyError: if no job store by the given alias is not found
+        """
+
+        try:
+            return self._jobstores[alias]
+        except KeyError:
+            raise KeyError('No such job store: %s' % alias)
+
+    def _lookup_job(self, job_id, jobstore_alias):
+        """
+        Finds a job by its ID.
+
+        :type job_id: str
+        :param str jobstore_alias: alias of a job store to look in
+        :return tuple[Job, str]: a tuple of job, jobstore alias (jobstore alias is None in case of a pending job)
+        :raises JobLookupError: if no job by the given ID is found.
+        """
+
+        # Check if the job is among the pending jobs
+        for job, alias, replace_existing in self._pending_jobs:
+            if job.id == job_id:
+                return job, None
+
+        # Look in all job stores
+        for alias, store in six.iteritems(self._jobstores):
+            if jobstore_alias in (None, alias):
+                job = store.lookup_job(job_id)
+                if job is not None:
+                    return job, alias
+
+        raise JobLookupError(job_id)
+
+    def _dispatch_event(self, event):
+        """
+        Dispatches the given event to interested listeners.
+
+        :param SchedulerEvent event: the event to send
+        """
+
+        with self._listeners_lock:
+            listeners = tuple(self._listeners)
+
+        for cb, mask in listeners:
+            if event.code & mask:
+                try:
+                    cb(event)
+                except:
+                    self._logger.exception('Error notifying listener')
+
+    def _real_add_job(self, job, jobstore_alias, replace_existing, wakeup):
+        """
+        :param Job job: the job to add
+        :param bool replace_existing: ``True`` to use update_job() in case the job already exists in the store
+        :param bool wakeup: ``True`` to wake up the scheduler after adding the job
+        """
+
+        # Fill in undefined values with defaults
+        replacements = {}
+        for key, value in six.iteritems(self._job_defaults):
+            if not hasattr(job, key):
+                replacements[key] = value
+
+        # Calculate the next run time if there is none defined
+        if not hasattr(job, 'next_run_time'):
+            now = datetime.now(self.timezone)
+            replacements['next_run_time'] = job.trigger.get_next_fire_time(None, now)
+
+        # Apply any replacements
+        job._modify(**replacements)
+
+        # Add the job to the given job store
+        store = self._lookup_jobstore(jobstore_alias)
+        try:
+            store.add_job(job)
+        except ConflictingIdError:
+            if replace_existing:
+                store.update_job(job)
+            else:
+                raise
+
+        # Mark the job as no longer pending
+        job._jobstore_alias = jobstore_alias
+
+        # Notify listeners that a new job has been added
+        event = JobEvent(EVENT_JOB_ADDED, job.id, jobstore_alias)
+        self._dispatch_event(event)
+
+        self._logger.info('Added job "%s" to job store "%s"', job.name, jobstore_alias)
+
+        # Notify the scheduler about the new job
+        if wakeup:
+            self.wakeup()
+
+    def _create_plugin_instance(self, type_, alias, constructor_kwargs):
+        """Creates an instance of the given plugin type, loading the plugin first if necessary."""
+
+        plugin_container, class_container, base_class = {
+            'trigger': (self._trigger_plugins, self._trigger_classes, BaseTrigger),
+            'jobstore': (self._jobstore_plugins, self._jobstore_classes, BaseJobStore),
+            'executor': (self._executor_plugins, self._executor_classes, BaseExecutor)
+        }[type_]
+
+        try:
+            plugin_cls = class_container[alias]
+        except KeyError:
+            if alias in plugin_container:
+                plugin_cls = class_container[alias] = plugin_container[alias].load()
+                if not issubclass(plugin_cls, base_class):
+                    raise TypeError('The {0} entry point does not point to a {0} class'.format(type_))
+            else:
+                raise LookupError('No {0} by the name "{1}" was found'.format(type_, alias))
+
+        return plugin_cls(**constructor_kwargs)
+
+    def _create_trigger(self, trigger, trigger_args):
+        if isinstance(trigger, BaseTrigger):
+            return trigger
+        elif trigger is None:
+            trigger = 'date'
+        elif not isinstance(trigger, six.string_types):
+            raise TypeError('Expected a trigger instance or string, got %s instead' % trigger.__class__.__name__)
+
+        # Use the scheduler's time zone if nothing else is specified
+        trigger_args.setdefault('timezone', self.timezone)
+
+        # Instantiate the trigger class
+        return self._create_plugin_instance('trigger', trigger, trigger_args)
+
+    def _create_lock(self):
+        """Creates a reentrant lock object."""
+
+        return RLock()
+
+    def _process_jobs(self):
+        """
+        Iterates through jobs in every jobstore, starts jobs that are due and figures out how long to wait for the next
+        round.
+        """
+
+        self._logger.debug('Looking for jobs to run')
+        now = datetime.now(self.timezone)
+        next_wakeup_time = None
+
+        with self._jobstores_lock:
+            for jobstore_alias, jobstore in six.iteritems(self._jobstores):
+                for job in jobstore.get_due_jobs(now):
+                    # Look up the job's executor
+                    try:
+                        executor = self._lookup_executor(job.executor)
+                    except:
+                        self._logger.error(
+                            'Executor lookup ("%s") failed for job "%s" -- removing it from the job store',
+                            job.executor, job)
+                        self.remove_job(job.id, jobstore_alias)
+                        continue
+
+                    run_times = job._get_run_times(now)
+                    run_times = run_times[-1:] if run_times and job.coalesce else run_times
+                    if run_times:
+                        try:
+                            executor.submit_job(job, run_times)
+                        except MaxInstancesReachedError:
+                            self._logger.warning(
+                                'Execution of job "%s" skipped: maximum number of running instances reached (%d)',
+                                job, job.max_instances)
+                        except:
+                            self._logger.exception('Error submitting job "%s" to executor "%s"', job, job.executor)
+
+                        # Update the job if it has a next execution time. Otherwise remove it from the job store.
+                        job_next_run = job.trigger.get_next_fire_time(run_times[-1], now)
+                        if job_next_run:
+                            job._modify(next_run_time=job_next_run)
+                            jobstore.update_job(job)
+                        else:
+                            self.remove_job(job.id, jobstore_alias)
+
+                # Set a new next wakeup time if there isn't one yet or the jobstore has an even earlier one
+                jobstore_next_run_time = jobstore.get_next_run_time()
+                if jobstore_next_run_time and (next_wakeup_time is None or jobstore_next_run_time < next_wakeup_time):
+                    next_wakeup_time = jobstore_next_run_time
+
+        # Determine the delay until this method should be called again
+        if next_wakeup_time is not None:
+            wait_seconds = max(timedelta_seconds(next_wakeup_time - now), 0)
+            self._logger.debug('Next wakeup is due at %s (in %f seconds)', next_wakeup_time, wait_seconds)
+        else:
+            wait_seconds = None
+            self._logger.debug('No jobs; waiting until a job is added')
+
+        return wait_seconds

lib/apscheduler/schedulers/blocking.py

@@ -0,0 +1,32 @@
+from __future__ import absolute_import
+from threading import Event
+
+from apscheduler.schedulers.base import BaseScheduler
+
+
+class BlockingScheduler(BaseScheduler):
+    """
+    A scheduler that runs in the foreground (:meth:`~apscheduler.schedulers.base.BaseScheduler.start` will block).
+    """
+
+    MAX_WAIT_TIME = 4294967  # Maximum value accepted by Event.wait() on Windows
+
+    _event = None
+
+    def start(self):
+        super(BlockingScheduler, self).start()
+        self._event = Event()
+        self._main_loop()
+
+    def shutdown(self, wait=True):
+        super(BlockingScheduler, self).shutdown(wait)
+        self._event.set()
+
+    def _main_loop(self):
+        while self.running:
+            wait_seconds = self._process_jobs()
+            self._event.wait(wait_seconds if wait_seconds is not None else self.MAX_WAIT_TIME)
+            self._event.clear()
+
+    def wakeup(self):
+        self._event.set()

lib/apscheduler/schedulers/gevent.py

@@ -0,0 +1,35 @@
+from __future__ import absolute_import
+
+from apscheduler.schedulers.blocking import BlockingScheduler
+from apscheduler.schedulers.base import BaseScheduler
+
+try:
+    from gevent.event import Event
+    from gevent.lock import RLock
+    import gevent
+except ImportError:  # pragma: nocover
+    raise ImportError('GeventScheduler requires gevent installed')
+
+
+class GeventScheduler(BlockingScheduler):
+    """A scheduler that runs as a Gevent greenlet."""
+
+    _greenlet = None
+
+    def start(self):
+        BaseScheduler.start(self)
+        self._event = Event()
+        self._greenlet = gevent.spawn(self._main_loop)
+        return self._greenlet
+
+    def shutdown(self, wait=True):
+        super(GeventScheduler, self).shutdown(wait)
+        self._greenlet.join()
+        del self._greenlet
+
+    def _create_lock(self):
+        return RLock()
+
+    def _create_default_executor(self):
+        from apscheduler.executors.gevent import GeventExecutor
+        return GeventExecutor()

lib/apscheduler/schedulers/qt.py

@@ -0,0 +1,46 @@
+from __future__ import absolute_import
+
+from apscheduler.schedulers.base import BaseScheduler
+
+try:
+    from PyQt5.QtCore import QObject, QTimer
+except ImportError:  # pragma: nocover
+    try:
+        from PyQt4.QtCore import QObject, QTimer
+    except ImportError:
+        try:
+            from PySide.QtCore import QObject, QTimer  # flake8: noqa
+        except ImportError:
+            raise ImportError('QtScheduler requires either PyQt5, PyQt4 or PySide installed')
+
+
+class QtScheduler(BaseScheduler):
+    """A scheduler that runs in a Qt event loop."""
+
+    _timer = None
+
+    def start(self):
+        super(QtScheduler, self).start()
+        self.wakeup()
+
+    def shutdown(self, wait=True):
+        super(QtScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._timer = QTimer.singleShot(wait_seconds * 1000, self._process_jobs)
+
+    def _stop_timer(self):
+        if self._timer:
+            if self._timer.isActive():
+                self._timer.stop()
+            del self._timer
+
+    def wakeup(self):
+        self._start_timer(0)
+
+    def _process_jobs(self):
+        wait_seconds = super(QtScheduler, self)._process_jobs()
+        self._start_timer(wait_seconds)

lib/apscheduler/schedulers/tornado.py

@@ -0,0 +1,60 @@
+from __future__ import absolute_import
+from datetime import timedelta
+from functools import wraps
+
+from apscheduler.schedulers.base import BaseScheduler
+from apscheduler.util import maybe_ref
+
+try:
+    from tornado.ioloop import IOLoop
+except ImportError:  # pragma: nocover
+    raise ImportError('TornadoScheduler requires tornado installed')
+
+
+def run_in_ioloop(func):
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        self._ioloop.add_callback(func, self, *args, **kwargs)
+    return wrapper
+
+
+class TornadoScheduler(BaseScheduler):
+    """
+    A scheduler that runs on a Tornado IOLoop.
+
+    =========== ===============================================================
+    ``io_loop`` Tornado IOLoop instance to use (defaults to the global IO loop)
+    =========== ===============================================================
+    """
+
+    _ioloop = None
+    _timeout = None
+
+    def start(self):
+        super(TornadoScheduler, self).start()
+        self.wakeup()
+
+    @run_in_ioloop
+    def shutdown(self, wait=True):
+        super(TornadoScheduler, self).shutdown(wait)
+        self._stop_timer()
+
+    def _configure(self, config):
+        self._ioloop = maybe_ref(config.pop('io_loop', None)) or IOLoop.current()
+        super(TornadoScheduler, self)._configure(config)
+
+    def _start_timer(self, wait_seconds):
+        self._stop_timer()
+        if wait_seconds is not None:
+            self._timeout = self._ioloop.add_timeout(timedelta(seconds=wait_seconds), self.wakeup)
+
+    def _stop_timer(self):
+        if self._timeout:
+            self._ioloop.remove_timeout(self._timeout)
+            del self._timeout
+
+    @run_in_ioloop
+    def wakeup(self):
+        self._stop_timer()
+        wait_seconds = self._process_jobs()
+        self._start_timer(wait_seconds)