Merge pull request #3367 from confor/fix-deps

Add imghdr and cgi

.gitignore

@@ -1,31 +1,48 @@
+[Tt]est[Rr]esult*
+/cache
+/logs
+.project
+.pydevproject
 
-# Compiled source #
-###################
+# coverage generated
+/cover-html/
+.coverage
+.coveralls.yml
+
+# Compiled source
 *.pyc
 *.py~
 *.pyproj
 *.sln
 
-# Logs and databases #
-######################
+# Headphones files
 *.log
 *.db*
 *.db-journal
 *.ini
+version.lock
 logs/*
 cache/*
 
-# OS generated files #
-######################
+# HTTPS Cert/Key
+*.crt
+*.key
+*.csr
+
+# OS generated files
 .DS_Store?
 .DS_Store
 ehthumbs.db
 Icon?
 Thumbs.db
 
-#ignore thumbnails created by windows
-Thumbs.db
-#Ignore files build by Visual Studio
+# Ignore files generated by PyCharm
+.idea/*
+
+# Ignore files generated by vi
+*.swp
+
+# Ignore files build by Visual Studio
 *.obj
 *.exe
 *.pdb
@@ -50,8 +67,7 @@ Thumbs.db
 obj/
 [Rr]elease*/
 _ReSharper*/
-[Tt]est[Rr]esult*
-/cache
-/logs
-.project
-.pydevproject
+.vscode
+
+# Python virtual env
+venv

.pep8

@@ -0,0 +1,7 @@
+[pep8]
+# E126 continuation line over-indented for hanging indent
+# E127 continuation line over-indented for visual indent
+# E128 continuation line under-indented for visual indent
+# E501 line too long (312 > 160 characters)
+ignore = E123,E126,E127,E128,E501
+max-line-length = 160

.travis.yml

@@ -0,0 +1,25 @@
+# Travis CI configuration file
+# http://about.travis-ci.org/docs/
+
+language: python
+
+sudo: false
+
+cache:
+  pip: true
+  directories:
+    - lib
+
+python:
+  - "2.7"
+
+install:
+  - pip install -r requirements-dev.txt
+
+script:
+  - pep8 headphones
+  - pyflakes headphones
+  - nosetests
+
+after_success:
+  - if [[ $TRAVIS_PYTHON_VERSION == "2.7" ]]; then coveralls; fi

API.md

@@ -0,0 +1,111 @@
+# API Reference
+The API is still pretty new and needs some serious cleaning up on the backend but should be reasonably functional. There are no error codes yet.
+
+## General structure
+The API endpoint is `http://ip:port + HTTP_ROOT + /api?apikey=$apikey&cmd=$command`
+
+Data response in JSON formatted. If executing a command like "delArtist" or "addArtist" you'll get back an "OK", else, you'll get the data you requested.
+
+## API methods
+
+### getIndex
+Fetch data from index page. Returns: ArtistName, ArtistSortName, ArtistID, Status, DateAdded, [LatestAlbum, ReleaseDate, AlbumID], HaveTracks, TotalTracks, IncludeExtras, LastUpdated, [ArtworkURL, ThumbURL]: a remote url to the artwork/thumbnail.
+
+To get the cached image path, see getArtistArt command. ThumbURL is added/updated when an artist is added/updated. If your using the database method to get the artwork, it's more reliable to use the ThumbURL than the ArtworkURL)
+
+### getArtist&id=$artistid
+Fetch artist data. returns the artist object (see above) and album info: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type, ArtworkURL: hosted image path. For cached image, see getAlbumArt command)
+
+### getAlbum&id=$albumid
+Fetch data from album page. Returns the album object, a description object and a tracks object. Tracks contain: AlbumASIN, AlbumTitle, TrackID, Format, TrackDuration (ms), ArtistName, TrackTitle, AlbumID, ArtistID, Location, TrackNumber, CleanName (stripped of punctuation /styling), BitRate)
+
+### getUpcoming
+Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type
+
+### getWanted
+Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type
+
+### getSimilar
+Returns similar artists  - with a higher "Count" being more likely to be similar. Returns: Count, ArtistName, ArtistID
+
+### getHistory
+Returns: Status, DateAdded, Title, URL (nzb), FolderName, AlbumID, Size (bytes)
+
+### getLogs
+Not working yet
+
+### findArtist&name=$artistname[&limit=$limit]
+Perform artist query on musicbrainz. Returns: url, score, name, uniquename (contains disambiguation info), id)
+
+### findAlbum&name=$albumname[&limit=$limit]
+Perform album query on musicbrainz. Returns: title, url (artist), id (artist), albumurl, albumid, score, uniquename (artist - with disambiguation)
+
+### addArtist&id=$artistid
+Add an artist to the db by artistid)
+
+### addAlbum&id=$releaseid
+Add an album to the db by album release id
+
+### delArtist&id=$artistid
+Delete artist from db by artistid)
+
+### pauseArtist&id=$artistid
+Pause an artist in db)
+
+### resumeArtist&id=$artistid
+Resume an artist in db)
+
+### refreshArtist&id=$artistid
+Refresh info for artist in db from musicbrainz
+
+### queueAlbum&id=$albumid[&new=True&lossless=True]
+Mark an album as wanted and start the searcher. Optional paramters: 'new' looks for new versions, 'lossless' looks only for lossless versions
+
+### unqueueAlbum&id=$albumid
+Unmark album as wanted / i.e. mark as skipped
+
+### forceSearch
+force search for wanted albums - not launched in a separate thread so it may take a bit to complete
+### forceProcess[&dir=/path/to/folder]
+Force post process albums in download directory - also not launched in a separate thread
+### forceActiveArtistsUpdate
+force Active Artist Update - also not launched in a separate thread
+
+### getVersion
+Returns some version information: git_path, install_type, current_version, installed_version, commits_behind
+
+### checkGithub
+Updates the version information above and returns getVersion data
+
+### shutdown
+Shut down headphones
+
+### restart
+Restart headphones
+
+### update
+Update headphones - you may want to check the install type in get version and not allow this if type==exe
+
+### getArtistArt&id=$artistid
+Returns either a relative path to the cached image, or a remote url if the image can't be saved to the cache dir
+
+getAlbumArt&id=$albumid
+see above
+
+### getArtistInfo&id=$artistid
+Returns Summary and Content, both formatted in html.
+
+### getAlbumInfo&id=$albumid
+See above, returns Summary and Content.
+
+### getArtistThumb&id=$artistid
+Returns either a relative path to the cached thumbnail artist image, or an http:// address if the cache dir can't be written to.
+
+### getAlbumThumb&id=$albumid
+See above.
+
+### choose_specific_download&id=$albumid
+Gives you a list of results from searcher.searchforalbum(). Basically runs a normal search, but rather than sorting them and downloading the best result, it dumps the data, which you can then pass on to download_specific_release(). Returns a list of dictionaries with params: title, size, url, provider & kind - all of these values must be passed back to download_specific_release
+
+### download_specific_release&id=albumid&title=$title&size=$size&url=$url&provider=$provider&kind=$kind
+Allows you to manually pass a choose_specific_download release back to searcher.send_to_downloader()

API_REFERENCE

@@ -1,67 +0,0 @@
-The API is still pretty new and needs some serious cleaning up on the backend but should be
-reasonably functional. There are no error codes yet, 
-
-
-General structure:
-http://localhost:8181 + HTTP_ROOT + /api?apikey=$apikey&cmd=$command
-
-Data returned in json format. If executing a command like "delArtist" or "addArtist" you'll get back an "OK", else, you'll get the data you requested
-
-$commands&parameters[&optionalparameters]:
-
-
-getIndex (fetch data from index page. Returns: ArtistName, ArtistSortName, ArtistID, Status, DateAdded,
-												[LatestAlbum, ReleaseDate, AlbumID], HaveTracks, TotalTracks,
-												IncludeExtras, LastUpdated, [ArtworkURL, ThumbURL]: a remote url to the artwork/thumbnail. To get the cached image path, see getArtistArt command.
-                                                                                                    ThumbURL is added/updated when an artist is added/updated. If your using the database method to get the artwork,
-                                                                                                    it's more reliable to use the ThumbURL than the ArtworkURL)
-
-getArtist&id=$artistid (fetch artist data. returns the artist object (see above) and album info: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type, ArtworkURL: hosted image path. For cached image, see getAlbumArt command)
-
-getAlbum&id=$albumid (fetch data from album page. Returns the album object, a description object and a tracks object. Tracks contain: AlbumASIN, AlbumTitle, TrackID, Format, TrackDuration (ms), ArtistName, TrackTitle, AlbumID, ArtistID, Location, TrackNumber, CleanName (stripped of punctuation /styling), BitRate)
-
-getUpcoming (Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type)
-
-getWanted (Returns: Status, AlbumASIN, DateAdded, AlbumTitle, ArtistName, ReleaseDate, AlbumID, ArtistID, Type)
-
-getSimilar (Returns similar artists  - with a higher "Count" being more likely to be similar. Returns: Count, ArtistName, ArtistID)
-
-getHistory (Returns: Status, DateAdded, Title, URL (nzb), FolderName, AlbumID, Size (bytes))
-
-getLogs (not working yet)
-
-findArtist&name=$artistname[&limit=$limit] (perform artist query on musicbrainz. Returns: url, score, name, uniquename (contains disambiguation info), id)
-
-findAlbum&name=$albumname[&limit=$limit]  (perform album query on musicbrainz. Returns: title, url (artist), id (artist), albumurl, albumid, score, uniquename (artist - with disambiguation)
-
-addArtist&id=$artistid (add an artist to the db by artistid)
-addAlbum&id=$releaseid (add an album to the db by album release id)
-
-delArtist&id=$artistid (delete artist from db by artistid)
-
-pauseArtist&id=$artistid (pause an artist in db)
-resumeArtist&id=$artistid (resume an artist in db)
-
-refreshArtist&id=$artistid (refresh info for artist in db from musicbrainz)
-
-queueAlbum&id=$albumid[&new=True&lossless=True] (Mark an album as wanted and start the searcher. Optional paramters: 'new' looks for new versions, 'lossless' looks only for lossless versions)
-unqueueAlbum&id=$albumid (Unmark album as wanted / i.e. mark as skipped)
-
-forceSearch (force search for wanted albums - not launched in a separate thread so it may take a bit to complete)
-forceProcess (force post process albums in download directory - also not launched in a separate thread)
-
-getVersion (Returns some version information: git_path, install_type, current_version, installed_version, commits_behind
-checkGithub (updates the version information above and returns getVersion data)
-
-shutdown (shut down headphones)
-restart (restart headphones)
-update (update headphones - you may want to check the install type in get version and not allow this if type==exe)
-
-getArtistArt&id=$artistid  (Returns either a relative path to the cached image, or a remote url if the image can't be saved to the cache dir)
-getAlbumArt&id=$albumid (see above)
-
-getArtistInfo&id=$artistid (Returns Summary and Content, both formatted in html)
-getAlbumInfo&id=$albumid (See above, returns Summary and Content)
-
-getArtistThumb&id=$artistid (Returns either a relative path to the cached thumbnail artist image, or an http:// address if the cache dir can't be written to)
-getAlbumThumb&id=$albumid (see above)

CHANGELOG.md

@@ -0,0 +1,414 @@
+# Changelog
+
+## v0.6.3
+ Released 26 May 2024
+
+Highlights:
+* Hotfix for searcher not returning results
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.6.2...v0.6.3).
+
+## v0.6.2
+ Released 26 May 2024
+
+Highlights:
+* Added soulseek support
+* Added bandcamp support
+* Changes and dependency updates to work with Python >= 3.12
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.6.1...v0.6.2).
+
+## v0.6.1
+R eleased 26 November 2023
+
+Highlights:
+* Dependency updates to work with > Python 3.11
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.6.0...v0.6.1).
+
+## v0.6.0
+Released 13 November 2022
+
+Highlights:
+* Updated to python 3
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.20...v0.6.0).
+
+## v0.5.20
+Released 15 October 2021
+
+Highlights:
+* Improved: Last.fm & metacrtic api fixes
+* Improved: Various database fixes
+* Added: qBitTorrent
+* Added: Telegram notifications
+* Improved: Many more since the last release. Check the list
+
+Many thanks to @AdeHub for keeping the ship afloat
+and to everyone else who contributed.
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.19...v0.5.20).
+
+## v0.5.19
+Released 27 March 2018
+
+Highlights:
+* Improved: Windows encoding fixes
+* Improved: Performance enhancements
+* Improved: Many more since the last release. Check the list
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.18...v0.5.19).
+
+## v0.5.18
+Released 01 December 2016
+
+Highlights:
+* Added: PassTheHeadphones support
+* Fixed: Special characters in password fields breaking on config page
+* Improved: Updated t411 url
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.17...v0.5.18).
+
+
+## v0.5.17
+Released 10 November 2016
+
+Highlights:
+* Added: t411 support
+* Fixed: Rutracker login
+* Fixed: Deluge empty password
+* Fixed: FreeBSD init script
+* Improved: Musicbrainz searching
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.16...v0.5.17).
+
+## v0.5.16
+Released 10 June 2016
+
+Hotfix update
+
+Highlights:
+* Fixed: Waffles url
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.15...v0.5.16).
+
+## v0.5.15
+Released 07 June 2016
+
+Hotfix update
+
+Highlights:
+* Fixed: Update vip mirror url to point to new server
+* Fixed: Update waffles url to .ch
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.14...v0.5.15).
+
+## v0.5.14
+Released 02 June 2016
+
+Highlights:
+* Fixed: File/folder format on new installs
+* Fixed: Pep8 errors
+* Improved: Updated fontawesome
+* Improved: Reverted back to less
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.13...v0.5.14).
+
+## v0.5.13
+Released 25 February 2016
+
+Another hotfix update
+
+Highlights:
+* Fixed: Saving config with non-defined options
+* Fixed: Pep8 errors
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.12...v0.5.13).
+
+## v0.5.12
+Released 25 February 2016
+
+This is mostly a hotfix update
+
+Highlights:
+* Added: Experimental Deluge Support
+* Fixed: Some pep8 stuff
+* Improved: Use curly braces for pathrender optional variables
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.11...v0.5.12).
+
+## v0.5.11
+Released 20 February 2016
+
+Highlights:
+* Added: Soft chroot option
+* Fixed: Post processing temporary directory fix (#2504)
+* Fixed: Ubuntu init script (#2509)
+* Fixed: Image cache uncaught exception (#2485)
+* Improved: $Date/$date variable in folder renaming
+* Improved: Reuse transmission session id
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.10...v0.5.11).
+
+## v0.5.10
+Released 29 January 2016
+
+Highlights:
+* Added: API option to post-process single folders
+* Added: Ability to specify extension when re-encoding
+* Added: Option to stop renaming folders
+* Fixed: Utorrent torrents not being removed (#2385)
+* Fixed: Torznab to transmission
+* Fixed: Magnet folder names in history
+* Fixed: Multiple torcache fixes
+* Fixed: Updated requests & urllib3 to latest versions to fix errors with pyOpenSSL
+* Improved: Use a temporary folder during post-processing
+* Improved: Added verify_ssl_cert option
+* Improved: Fixed track matching progress
+* Improved: pylint, pep8 & pylint fixes
+* Improved: Stop JS links from scrolling to the top of the page
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.9...v0.5.10).
+
+## v0.5.9
+Released 05 September 2015
+
+Highlights:
+* Added: Providers Strike, Jackett, custom Torznabs
+* Added: Option to stop post-processing if no good match found (#2343)
+* Fixed: Blackhole -> Magnet, limit to torcache
+* Fixed: Kat 403 flac error
+* Fixed: Last.fm errors
+* Fixed: Pushover notifications
+* Improved: Rutracker logging, switched to requests lib
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.8...v0.5.9).
+
+## v0.5.8
+Released 13 July 2015
+
+Highlights:
+* Added: Option to only include official extras
+* Added: Option to wait until album release date before searching
+* Fixed: NotifyMyAndroid notifications
+* Fixed: Plex Notifications
+* Fixed: Metacritic parsing
+* Fixed: Pushbullet notifications
+* Fixed: What.cd not honoring custom search term (#2279)
+* Improved: XSS Search bug
+* Improved: Config page layout
+* Improved: Set localhost as default
+* Improved: Better single artist scanning
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.7...v0.5.8).
+
+## v0.5.7
+Released 01 July 2015
+
+Highlights:
+* Improved: Moved pushover to use requests lib
+* Improved: Plex tokens with Plex Home
+* Improved: Added getLogs & clearLogs to api
+* Improved: Cache MetaCritic scores. Added user-agent header to fix 403 errors
+* Improved: Specify whether to delete folders when force post-processing
+* Improved: Convert target bitrate to vbr preset for what.cd searching
+* Improved: Switched Pushover to requests lib
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.6...v0.5.7).
+
+## v0.5.6
+Released 08 June 2015
+
+Highlights:
+* Added: Metacritic scores
+* Added: Series support (e.g. Cafe Del Mar, Now That's What I Call Music, etc)
+* Added: Filter out clean/edited/censored releases (#2198)
+* Added: Button on the log page to toggle verbose/debug logging
+* Fixed: Connecting to SABnzbd over https with python >= 2.7.9
+* Fixed: Email Notifications with SSL
+* Fixed: Don't limit musicbrainz results to first 100
+* Fixed: nzbget url fix
+* Fixed: OSX Notifications
+* Improved: Cuesplit, allow wav, ape to be split
+* Improved: Moved the 'freeze db' option to the advanced->misc. tab
+* Improved: Moved kickass searching to json api, so it doesn't throw 404 errors anymore when there are no results
+* Improved: SSL for headphones indexer
+* Improved: Disable update dialog box if check_github is diabled
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.5...v0.5.6).
+
+## v0.5.5
+Released 04 May 2015
+
+Highlights:
+* Added: force ID3v2.3 during post processing (#2121)
+* Added: MusicBrainz authentication (#2125)
+* Added: Email notifications (addresses #1045)
+* Fixed: Kickass url updated to kickass.to (#2119)
+* Fixed: Piratebay returning 0 bytes for all files
+* Fixed: Albums stopped automatically refreshing when adding an artist
+* Fixed: Min/max sizes for target bitrate
+* Fixed: Don't filter any results if looking for a specific download
+* Fixed: Sort by size in the specific download table
+* Fixed: Deal with beets recommendation.none correctly
+* Improved: Close dialog window automatically when choosing a specific download
+* Improved: Move some repetitive log messages to debug level
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.4...v0.5.5).
+
+## v0.5.4
+Released 05 February 2015
+
+Highlights:
+* Added: backported 'Scan Artist' feature from `sarakha63/headphones`
+* Fixed: change file permissions of cache files according to settings (#2102)
+* Fixed: hide Songkick if no regional events are available
+* Fixed: only reschedule jobs if timeout changed (#2099)
+* Fixed: limit dialog height (#2096)
+* Improved: upgraded requests to 2.5.1
+* Improved: upgraded unidecode to 0.04.17
+* Improved: upgraded tzlocal to 1.1.2
+* Improved: upgraded PyTZ to 2014.10
+* Improved: upgraded mako to 1.0.1
+* Improved: upgraded mutagen to 1.27
+* Improved: upgraded beets to 1.3.10
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.3...v0.5.4).
+
+## v0.5.3
+Released 15 January 2015
+
+Highlights:
+* Added: update active artists via API (#2075)
+* Fixed: queue instance lacks `add` method (#2055)
+* Improved: better SSL error messages (#2058)
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.2...v0.5.3).
+
+## v0.5.2
+Released 28 December 2014
+
+Highlights:
+* Added: advanced option to ignore certain folders by patterns. (#2037)
+* Added: advanced option to ignore certain files by patterns (library only)
+* Added: specify optional paths to CUE splitting tools (#1938)
+* Added: experimental support for OPB (provide details yourself!)
+* Fixed: magnet to torrent conversion (#1926)
+* Fixed: new KAT URL (#2043)
+* Fixed: LMS notifications (#1564)
+* Improved: notify user of SSL-related warnings, instead of silently failing.
+* Improved: show all search results for 'Choose Specific Release'
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5.1...v0.5.2).
+
+## v0.5.1
+Released 13 November 2014
+
+Highlights:
+* Added: allow one to disable interval tasks (#2002)
+* Added: script to downgrade Headphones to last version that started (Linux only)
+* Fixed: SSL issues in CherryPy. Self-generated certificates will be 2048 now (#1995)
+* Fixed: Transmission URL detection (#1998)
+* Fixed: missing dependencies for APScheduler and CherryPy (#2001)
+* Improved: symlink infinite recursion detection
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.5...v0.5.1).
+
+## v0.5
+Released 10 November 2014
+
+Highlights:
+* Added: CUE splitter
+* Added: filter search result by MusicBrainz Release Group ID
+* Added: follow symlinks while scanning library (#1953)
+* Fixed: crash during post processing (#1897)
+* Fixed: embedding lyrics (#1896)
+* Fixed: HTTP errors with older versions of Python 2.6
+* Fixed: jump back to top of page (#1948)
+* Improved: parse MusicBrainz RGID first when post processing (#1952)
+* Improved: Growl unicode characters (#1695)
+* Improved: search handling for PB and KAT
+* Improved: Last.FM API support (#1877)
+* Improved: upgraded CherryPy to version 3.6.0
+* Improved: upgraded Requests to version 2.4.1
+* Improved: upgraded APScheduler to version 3.0.1
+* Improved: lot of code refactoring
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.4...v0.5).
+
+## v0.4
+Released 20 September 2014
+
+Highlights:
+* Added: support for libav-tools (which replaces FFmpeg under Ubuntu)
+* Added: option to freeze library when post processing
+* Added: Songkick per area
+* Added: rename original NFO file (#1797)
+* Removed: removed dead search providers
+* Fixed: removed left-overs of old packages
+* Improved: rename by original folder (#1811)
+* Improved: uTorrent fixess
+* Improved: delayed loading of album art
+* Improved: search result parser
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.4...v0.4).
+
+## v0.3.4
+Released 15 May 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.3...v0.3.4).
+
+## v0.3.3
+Released 05 May 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.2...v0.3.3).
+
+## v0.3.2
+Released 17 April 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3.1...v0.3.2).
+
+## v0.3.1
+Released 11 April 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.3...v0.3.1).
+
+## v0.3
+Released 09 April 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2.3...v0.3).
+
+## v0.2.3
+Released 28 March 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2.2...v0.2.3).
+
+## v0.2.2
+Released 15 January 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2.1...v0.2.2).
+
+## v0.2.1
+Released 13 January 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.2...v0.2.1).
+
+## v0.2
+Released 13 January 2014
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.1.2...v0.2).
+
+## v0.1.2
+Released 11 October 2013
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.1.1...v0.1.2).
+
+## v0.1.1
+Released 26 August 2013
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/v0.1...v0.1.1).
+
+## v0.1
+Released 05 August 2013
+
+The full list of commits can be found [here](https://github.com/rembo10/headphones/compare/2156e1341405d07c5bcfbe994f6b354b32d94cda...v0.1).

CONTRIBUTING.md

@@ -0,0 +1,52 @@
+# Contributing to Headphones
+
+## Issues
+In case you read this because you are posting an issue, please take a minute and conside the things below. The issue tracker is not a support forum. It is primarily intended to submit bugs, improvements or feature requests. However, we are glad to help you, and make sure the problem is not caused by Headphones, but don't expect step-by-step answers.
+
+* Use the search function. Chances are that your problem is already discussed. Do not append to (closed) issues if your problem does not fit the discussion.
+* Visit the [Troubleshooting](../../wiki/TroubleShooting) wiki first.
+* Use [proper formatting](https://help.github.com/articles/github-flavored-markdown/). Paste your logs in code blocks.
+* Close your issue if you resolved it.
+
+## Pull Requests
+If you think you can contribute code to the Headphones repository, do not hesitate to submit a pull request.
+
+### Branches
+All pull requests should be based on the `develop` branch, to minimize cross merges. When you want to develop a new feature, clone the repository with `git clone origin/develop -b FEATURE_NAME`. Use meaningful commit messages.
+
+### Python Code
+
+#### Compatibility
+The code should work with Python 2.6 and 2.7. Note that Headphones runs on different platforms, including Network Attached Storage devices such as Synology.
+
+Re-use existing code. Do not hesitate to add logging in your code. You can the logger module `headphones.logger.*` for this. Web requests are invoked via `headphones.request.*` and derived ones. Use these methods to automatically add proper and meaningful error handling.
+
+#### Code conventions
+Although Headphones did not adapt a code convention in the past, we try to follow the [PEP8](http://legacy.python.org/dev/peps/pep-0008/) conventions for future code. A short summary to remind you (copied from http://wiki.ros.org/PyStyleGuide):
+
+ * 4 space indentation
+ * 80 characters per line
+ * `package_name`
+ * `ClassName`
+ * `method_name`
+ * `field_name`
+ * `_private_something`
+ * `self.__really_private_field`
+ * `_global`
+
+#### Documentation
+Document your code. Use docstrings See [PEP-257](https://www.python.org/dev/peps/pep-0257/) for more information.
+
+#### Continuous Integration
+Headphones has a configuration file for [travis-ci](https://travis-ci.org/). You can add your forked repo to Travis to have it check your code against PEP8, PyLint, and PyFlakes for you. Your pull request will show a green check mark or a red cross on each tested commit, depending on if linting passes.
+
+### HTML/Template code
+
+#### Compatibility
+HTML5 compatible browsers are targetted. There is no specific mobile version of Headphones yet.
+
+#### Conventions
+* 4 space indentation
+* `methodName`
+* `variableName`
+* `ClassName`

Headphones.py

@@ -14,26 +14,34 @@
 #  You should have received a copy of the GNU General Public License
 #  along with Headphones.  If not, see <http://www.gnu.org/licenses/>.
 
-import os, sys, locale
-import time
-import signal
+import os
+import sys
 
-from lib.configobj import ConfigObj
+if sys.version_info <= (3, 6):
+    sys.stdout.write("Headphones requires Python >= 3.7\n")
+    sys.exit(1)
 
-import headphones
+# Ensure lib added to path, before any other imports
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib/'))
 
 from headphones import webstart, logger
 
-try:
-    import argparse
-except ImportError:
-    import lib.argparse as argparse
+import locale
+import time
+import signal
+import argparse
+import headphones
 
+# Register signals, such as CTRL + C
 signal.signal(signal.SIGINT, headphones.sig_handler)
 signal.signal(signal.SIGTERM, headphones.sig_handler)
 
 
 def main():
+    """
+    Headphones application entry point. Parses arguments, setups encoding and
+    initializes the application.
+    """
 
     # Fixed paths to Headphones
     if hasattr(sys, 'frozen'):
@@ -50,7 +58,10 @@ def main():
 
     try:
         locale.setlocale(locale.LC_ALL, "")
-        headphones.SYS_ENCODING = locale.getpreferredencoding()
+        if headphones.SYS_PLATFORM == 'win32':
+            headphones.SYS_ENCODING = sys.getdefaultencoding().upper()
+        else:
+            headphones.SYS_ENCODING = locale.getpreferredencoding()
     except (locale.Error, IOError):
         pass
 
@@ -59,111 +70,155 @@ def main():
         headphones.SYS_ENCODING = 'UTF-8'
 
     # Set up and gather command line arguments
-    parser = argparse.ArgumentParser(description='Music add-on for SABnzbd+')
+    parser = argparse.ArgumentParser(
+        description='Music add-on for SABnzbd+, Transmission and more.')
 
-    parser.add_argument('-v', '--verbose', action='store_true', help='Increase console logging verbosity')
-    parser.add_argument('-q', '--quiet', action='store_true', help='Turn off console logging')
-    parser.add_argument('-d', '--daemon', action='store_true', help='Run as a daemon')
-    parser.add_argument('-p', '--port', type=int, help='Force Headphones to run on a specified port')
-    parser.add_argument('--datadir', help='Specify a directory where to store your data files')
+    parser.add_argument(
+        '-v', '--verbose', action='store_true', help='Increase console logging verbosity')
+    parser.add_argument(
+        '-q', '--quiet', action='store_true', help='Turn off console logging')
+    parser.add_argument(
+        '-d', '--daemon', action='store_true', help='Run as a daemon')
+    parser.add_argument(
+        '-p', '--port', type=int, help='Force Headphones to run on a specified port')
+    parser.add_argument(
+        '--datadir', help='Specify a directory where to store your data files')
     parser.add_argument('--config', help='Specify a config file to use')
-    parser.add_argument('--nolaunch', action='store_true', help='Prevent browser from launching on startup')
-    parser.add_argument('--pidfile', help='Create a pid file (only relevant when running as a daemon)')
+    parser.add_argument('--nolaunch', action='store_true',
+                        help='Prevent browser from launching on startup')
+    parser.add_argument(
+        '--pidfile', help='Create a pid file (only relevant when running as a daemon)')
+    parser.add_argument(
+        '--host', help='Specify a host (default - localhost)')
 
     args = parser.parse_args()
 
     if args.verbose:
-        headphones.VERBOSE = 2
-    elif args.quiet:
-        headphones.VERBOSE = 0
+        headphones.VERBOSE = True
+    if args.quiet:
+        headphones.QUIET = True
+
+    # Do an intial setup of the logger.
+    logger.initLogger(console=not headphones.QUIET, log_dir=False,
+        verbose=headphones.VERBOSE)
 
     if args.daemon:
         if sys.platform == 'win32':
-            print "Daemonize not supported under Windows, starting normally"
+            sys.stderr.write(
+                "Daemonizing not supported under Windows, starting normally\n")
         else:
-            headphones.DAEMON=True
-            headphones.VERBOSE = False
+            headphones.DAEMON = True
+            headphones.QUIET = True
 
     if args.pidfile:
         headphones.PIDFILE = str(args.pidfile)
 
-        # If the pidfile already exists, headphones may still be running, so exit
+        # If the pidfile already exists, headphones may still be running, so
+        # exit
         if os.path.exists(headphones.PIDFILE):
-            sys.exit("PID file '" + headphones.PIDFILE + "' already exists. Exiting.")
+            raise SystemExit("PID file '%s' already exists. Exiting." %
+                headphones.PIDFILE)
 
-        # The pidfile is only useful in daemon mode, make sure we can write the file properly
+        # The pidfile is only useful in daemon mode, make sure we can write the
+        # file properly
         if headphones.DAEMON:
             headphones.CREATEPID = True
-            try:
-                file(headphones.PIDFILE, 'w').write("pid\n")
-            except IOError, e:
-                raise SystemExit("Unable to write PID file: %s [%d]" % (e.strerror, e.errno))
-        else:
-            logger.warn("Not running in daemon mode. PID file creation disabled.")
 
+            try:
+                with open(headphones.PIDFILE, 'w') as fp:
+                    fp.write("pid\n")
+            except IOError as e:
+                raise SystemExit("Unable to write PID file: %s", e)
+        else:
+            logger.warn("Not running in daemon mode. PID file creation " \
+                "disabled.")
+
+    # Determine which data directory and config file to use
     if args.datadir:
         headphones.DATA_DIR = args.datadir
     else:
         headphones.DATA_DIR = headphones.PROG_DIR
 
     if args.config:
-        headphones.CONFIG_FILE = args.config
+        config_file = args.config
     else:
-        headphones.CONFIG_FILE = os.path.join(headphones.DATA_DIR, 'config.ini')
+        config_file = os.path.join(headphones.DATA_DIR, 'config.ini')
 
     # Try to create the DATA_DIR if it doesn't exist
     if not os.path.exists(headphones.DATA_DIR):
         try:
             os.makedirs(headphones.DATA_DIR)
         except OSError:
-            raise SystemExit('Could not create data directory: ' + headphones.DATA_DIR + '. Exiting....')
+            raise SystemExit(
+                'Could not create data directory: ' + headphones.DATA_DIR + '. Exiting....')
 
     # Make sure the DATA_DIR is writeable
     if not os.access(headphones.DATA_DIR, os.W_OK):
-        raise SystemExit('Cannot write to the data directory: ' + headphones.DATA_DIR + '. Exiting...')
+        raise SystemExit(
+            'Cannot write to the data directory: ' + headphones.DATA_DIR + '. Exiting...')
 
     # Put the database in the DATA_DIR
     headphones.DB_FILE = os.path.join(headphones.DATA_DIR, 'headphones.db')
 
-    headphones.CFG = ConfigObj(headphones.CONFIG_FILE, encoding='utf-8')
-
-    # Read config & start logging
-    headphones.initialize()
+    # Read config and start logging
+    try:
+        headphones.initialize(config_file)
+    except headphones.exceptions.SoftChrootError as e:
+        raise SystemExit('FATAL ERROR')
 
     if headphones.DAEMON:
-        if sys.platform == "win32":
-            print "Daemonize not supported under Windows, starting normally"
-        else:
-            headphones.daemonize()
+        headphones.daemonize()
 
-    #configure the connection to the musicbrainz database
+    # Configure the connection to the musicbrainz database
     headphones.mb.startmb()
 
     # Force the http port if neccessary
     if args.port:
         http_port = args.port
-        logger.info('Starting Headphones on forced port: %i' % http_port)
+        logger.info('Using forced web server port: %i', http_port)
     else:
-        http_port = int(headphones.HTTP_PORT)
+        http_port = int(headphones.CONFIG.HTTP_PORT)
 
-    # Try to start the server.
-    webstart.initialize({
-                    'http_port':        http_port,
-                    'http_host':        headphones.HTTP_HOST,
-                    'http_root':        headphones.HTTP_ROOT,
-                    'http_proxy':       headphones.HTTP_PROXY,
-                    'http_username':    headphones.HTTP_USERNAME,
-                    'http_password':    headphones.HTTP_PASSWORD,
-            })
+    # Force the http host if neccessary
+    if args.host:
+        http_host = args.host
+        logger.info('Using forced web server host: %s', http_host)
+    else:
+        http_host = headphones.CONFIG.HTTP_HOST
 
-    logger.info('Starting Headphones on port: %i' % http_port)
+    # Check if pyOpenSSL is installed. It is required for certificate generation
+    # and for CherryPy.
+    if headphones.CONFIG.ENABLE_HTTPS:
+        try:
+            import OpenSSL
+        except ImportError:
+            logger.warn("The pyOpenSSL module is missing. Install this " \
+                "module to enable HTTPS. HTTPS will be disabled.")
+            headphones.CONFIG.ENABLE_HTTPS = False
 
-    if headphones.LAUNCH_BROWSER and not args.nolaunch:
-        headphones.launch_browser(headphones.HTTP_HOST, http_port, headphones.HTTP_ROOT)
+    # Try to start the server. Will exit here is address is already in use.
+    web_config = {
+        'http_port': http_port,
+        'http_host': http_host,
+        'http_root': headphones.CONFIG.HTTP_ROOT,
+        'http_proxy': headphones.CONFIG.HTTP_PROXY,
+        'enable_https': headphones.CONFIG.ENABLE_HTTPS,
+        'https_cert': headphones.CONFIG.HTTPS_CERT,
+        'https_key': headphones.CONFIG.HTTPS_KEY,
+        'http_username': headphones.CONFIG.HTTP_USERNAME,
+        'http_password': headphones.CONFIG.HTTP_PASSWORD,
+    }
+    webstart.initialize(web_config)
 
     # Start the background threads
     headphones.start()
 
+    # Open webbrowser
+    if headphones.CONFIG.LAUNCH_BROWSER and not args.nolaunch:
+        headphones.launch_browser(headphones.CONFIG.HTTP_HOST, http_port,
+                                  headphones.CONFIG.HTTP_ROOT)
+
+    # Wait endlessy for a signal to happen
     while True:
         if not headphones.SIGNAL:
             try:
@@ -171,7 +226,8 @@ def main():
             except KeyboardInterrupt:
                 headphones.SIGNAL = 'shutdown'
         else:
-            logger.info('Received signal: ' + headphones.SIGNAL)
+            logger.info('Received signal: %s', headphones.SIGNAL)
+
             if headphones.SIGNAL == 'shutdown':
                 headphones.shutdown()
             elif headphones.SIGNAL == 'restart':
@@ -181,7 +237,6 @@ def main():
 
             headphones.SIGNAL = None
 
-    return
-
+# Call main()
 if __name__ == "__main__":
     main()

README.md

@@ -1,44 +1,67 @@
-#![preview thumb](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png)Headphones
-
-###Installation and Notes
-
-This is a relatively early release of a third-party add-on for SABnzbd. It's been around for about a year, and while
-quite functional, there are still a lot of kinks to work out!
-
-To run it, just double-click the Headphones.py file (in Windows - you may need to right click and click 'Open With' -> Python) or launch a terminal, cd into the Headphones directory and run 'python headphones.py'.
-
-For additional startup options, type 'python Headphones.py -h' or 'python Headphones.py --help'
-
-###Screenshots
-
-Homepage (Artist Overview)
-
-![preview thumb](http://i.imgur.com/LZO9a.png)
-
-One of the many settings pages....
-
-![preview thumb](http://i.imgur.com/xcWNy.png)
-
-It might even know you better than you know yourself:
-
-![preview thumb](http://i.imgur.com/R7J0f.png)
-
-Import Your Favorite Artists:
-
-![preview thumb](http://i.imgur.com/6tZoC.png)
-
-Artist Search Results (also search by album!):
-
-![preview thumb](http://i.imgur.com/rIV0P.png)
-
-Artist Page with Bio & Album Overview:
-
-![preview thumb](http://i.imgur.com/SSil1.png)
-
-Album Page with track overview:
-
-![preview thumb](http://i.imgur.com/kcjES.png)
-
-If you run into any issues, use the GitHub issue tracker at: http://github.com/rembo10/headphones/issues. 
-
-This is free software under the GPL v3 open source license - so feel free to do with it what you wish.
+## ![Headphones Logo](https://github.com/rembo10/headphones/raw/master/data/images/headphoneslogo.png) Headphones
+
+[![Build Status](https://travis-ci.org/rembo10/headphones.svg?branch=master)](https://travis-ci.org/rembo10/headphones)
+[![Build Status](https://img.shields.io/travis/rembo10/headphones/develop.svg?label=develop%20branch%20build)](https://travis-ci.org/rembo10/headphones)
+
+Headphones is an automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, µTorrent, Deluge and Blackhole.
+
+## Support & Discuss
+You are free to join the Headphones support community on IRC where you can ask questions, hang around and discuss anything related to HP.
+
+1. Use any IRC client and connect to the Freenode server, `irc.freenode.net`.
+2. Join the `#headphones` channel.
+
+## Installation and Notes
+
+* [Installation page](../../wiki/Installation) shows you how to install Headphones.
+* [Usage guide](../../wiki/Usage-guide) introduces you to Headphones.
+* [Troubleshooting page](../../wiki/TroubleShooting) in the wiki can help you with common problems.
+
+**Issues** can be reported on the GitHub issue tracker considering these rules:
+
+1. Analyze your log, you just might find the solution yourself!
+2. You read the wiki and searched existing issues, but this is not solving your problem.
+3. Post the issue with a clear title, description and the HP log and use [proper markdown syntax](https://help.github.com/articles/github-flavored-markdown) to structure your text (code/log in code blocks).
+4. Close your issue when it's solved! If you found the solution yourself, please comment so that others benefit from it.
+
+**Feature requests** can be reported on the GitHub issue tracker too:
+
+1. Search for similar existing 'issues', feature requests can be recognized by the label 'Request'.
+2. If a similar Request exists, post a comment (+1, or add a new idea to the existing request), otherwise you can create a new one.
+
+If you **comply with these rules** you can [post your request/issue](http://github.com/rembo10/headphones/issues).
+
+**Support** the project by implementing new features, solving support tickets and provide bug fixes.
+
+## Screenshots
+
+Homepage (Artist Overview):
+
+![preview thumb](http://i.imgur.com/LZO9a.png)
+
+One of the many settings pages:
+
+![preview thumb](http://i.imgur.com/xcWNy.png)
+
+It might even know you better than you know yourself:
+
+![preview thumb](http://i.imgur.com/R7J0f.png)
+
+Import Your Favorite Artists:
+
+![preview thumb](http://i.imgur.com/6tZoC.png)
+
+Search Results:
+
+![preview thumb](http://i.imgur.com/rIV0P.png)
+
+Artist Page with Bio & Album Overview:
+
+![preview thumb](http://i.imgur.com/SSil1.png)
+
+Album Page with track overview:
+
+![preview thumb](http://i.imgur.com/kcjES.png)
+
+## License
+This is free software under the GPL v3 open source license. Feel free to do with it what you wish, but any modification must be open sourced. A copy of the license is included.

TODO

@@ -1,8 +0,0 @@
-##############################
-###   TODO LIST (18/08/12) ###
-##############################
-
-1. Add ability to fetch a specific extra, rather than pull down the whole category
-    - This requires extra calls to MB right now, so modify 1st mb artist call to pull down the
-      whole album list, then only send the desried ones to importer.py
-2. Update API with newstyle getExtras function (lacking entirely right now)

bs4/__init__.py

@@ -1,359 +0,0 @@
-"""Beautiful Soup
-Elixir and Tonic
-"The Screen-Scraper's Friend"
-http://www.crummy.com/software/BeautifulSoup/
-
-Beautiful Soup uses a pluggable XML or HTML parser to parse a
-(possibly invalid) document into a tree representation. Beautiful Soup
-provides provides methods and Pythonic idioms that make it easy to
-navigate, search, and modify the parse tree.
-
-Beautiful Soup works with Python 2.6 and up. It works better if lxml
-and/or html5lib is installed.
-
-For more than you ever wanted to know about Beautiful Soup, see the
-documentation:
-http://www.crummy.com/software/BeautifulSoup/bs4/doc/
-"""
-
-__author__ = "Leonard Richardson (leonardr@segfault.org)"
-__version__ = "4.1.3"
-__copyright__ = "Copyright (c) 2004-2012 Leonard Richardson"
-__license__ = "MIT"
-
-__all__ = ['BeautifulSoup']
-
-import re
-import warnings
-
-from .builder import builder_registry
-from .dammit import UnicodeDammit
-from .element import (
-    CData,
-    Comment,
-    DEFAULT_OUTPUT_ENCODING,
-    Declaration,
-    Doctype,
-    NavigableString,
-    PageElement,
-    ProcessingInstruction,
-    ResultSet,
-    SoupStrainer,
-    Tag,
-    )
-
-# The very first thing we do is give a useful error if someone is
-# running this code under Python 3 without converting it.
-syntax_error = u'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work. You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'
-
-class BeautifulSoup(Tag):
-    """
-    This class defines the basic interface called by the tree builders.
-
-    These methods will be called by the parser:
-      reset()
-      feed(markup)
-
-    The tree builder may call these methods from its feed() implementation:
-      handle_starttag(name, attrs) # See note about return value
-      handle_endtag(name)
-      handle_data(data) # Appends to the current data node
-      endData(containerClass=NavigableString) # Ends the current data node
-
-    No matter how complicated the underlying parser is, you should be
-    able to build a tree using 'start tag' events, 'end tag' events,
-    'data' events, and "done with data" events.
-
-    If you encounter an empty-element tag (aka a self-closing tag,
-    like HTML's <br> tag), call handle_starttag and then
-    handle_endtag.
-    """
-    ROOT_TAG_NAME = u'[document]'
-
-    # If the end-user gives no indication which tree builder they
-    # want, look for one with these features.
-    DEFAULT_BUILDER_FEATURES = ['html', 'fast']
-
-    # Used when determining whether a text node is all whitespace and
-    # can be replaced with a single space. A text node that contains
-    # fancy Unicode spaces (usually non-breaking) should be left
-    # alone.
-    STRIP_ASCII_SPACES = {9: None, 10: None, 12: None, 13: None, 32: None, }
-
-    def __init__(self, markup="", features=None, builder=None,
-                 parse_only=None, from_encoding=None, **kwargs):
-        """The Soup object is initialized as the 'root tag', and the
-        provided markup (which can be a string or a file-like object)
-        is fed into the underlying parser."""
-
-        if 'convertEntities' in kwargs:
-            warnings.warn(
-                "BS4 does not respect the convertEntities argument to the "
-                "BeautifulSoup constructor. Entities are always converted "
-                "to Unicode characters.")
-
-        if 'markupMassage' in kwargs:
-            del kwargs['markupMassage']
-            warnings.warn(
-                "BS4 does not respect the markupMassage argument to the "
-                "BeautifulSoup constructor. The tree builder is responsible "
-                "for any necessary markup massage.")
-
-        if 'smartQuotesTo' in kwargs:
-            del kwargs['smartQuotesTo']
-            warnings.warn(
-                "BS4 does not respect the smartQuotesTo argument to the "
-                "BeautifulSoup constructor. Smart quotes are always converted "
-                "to Unicode characters.")
-
-        if 'selfClosingTags' in kwargs:
-            del kwargs['selfClosingTags']
-            warnings.warn(
-                "BS4 does not respect the selfClosingTags argument to the "
-                "BeautifulSoup constructor. The tree builder is responsible "
-                "for understanding self-closing tags.")
-
-        if 'isHTML' in kwargs:
-            del kwargs['isHTML']
-            warnings.warn(
-                "BS4 does not respect the isHTML argument to the "
-                "BeautifulSoup constructor. You can pass in features='html' "
-                "or features='xml' to get a builder capable of handling "
-                "one or the other.")
-
-        def deprecated_argument(old_name, new_name):
-            if old_name in kwargs:
-                warnings.warn(
-                    'The "%s" argument to the BeautifulSoup constructor '
-                    'has been renamed to "%s."' % (old_name, new_name))
-                value = kwargs[old_name]
-                del kwargs[old_name]
-                return value
-            return None
-
-        parse_only = parse_only or deprecated_argument(
-            "parseOnlyThese", "parse_only")
-
-        from_encoding = from_encoding or deprecated_argument(
-            "fromEncoding", "from_encoding")
-
-        if len(kwargs) > 0:
-            arg = kwargs.keys().pop()
-            raise TypeError(
-                "__init__() got an unexpected keyword argument '%s'" % arg)
-
-        if builder is None:
-            if isinstance(features, basestring):
-                features = [features]
-            if features is None or len(features) == 0:
-                features = self.DEFAULT_BUILDER_FEATURES
-            builder_class = builder_registry.lookup(*features)
-            if builder_class is None:
-                raise FeatureNotFound(
-                    "Couldn't find a tree builder with the features you "
-                    "requested: %s. Do you need to install a parser library?"
-                    % ",".join(features))
-            builder = builder_class()
-        self.builder = builder
-        self.is_xml = builder.is_xml
-        self.builder.soup = self
-
-        self.parse_only = parse_only
-
-        self.reset()
-
-        if hasattr(markup, 'read'):        # It's a file-type object.
-            markup = markup.read()
-        (self.markup, self.original_encoding, self.declared_html_encoding,
-         self.contains_replacement_characters) = (
-            self.builder.prepare_markup(markup, from_encoding))
-
-        try:
-            self._feed()
-        except StopParsing:
-            pass
-
-        # Clear out the markup and remove the builder's circular
-        # reference to this object.
-        self.markup = None
-        self.builder.soup = None
-
-    def _feed(self):
-        # Convert the document to Unicode.
-        self.builder.reset()
-
-        self.builder.feed(self.markup)
-        # Close out any unfinished strings and close all the open tags.
-        self.endData()
-        while self.currentTag.name != self.ROOT_TAG_NAME:
-            self.popTag()
-
-    def reset(self):
-        Tag.__init__(self, self, self.builder, self.ROOT_TAG_NAME)
-        self.hidden = 1
-        self.builder.reset()
-        self.currentData = []
-        self.currentTag = None
-        self.tagStack = []
-        self.pushTag(self)
-
-    def new_tag(self, name, namespace=None, nsprefix=None, **attrs):
-        """Create a new tag associated with this soup."""
-        return Tag(None, self.builder, name, namespace, nsprefix, attrs)
-
-    def new_string(self, s):
-        """Create a new NavigableString associated with this soup."""
-        navigable = NavigableString(s)
-        navigable.setup()
-        return navigable
-
-    def insert_before(self, successor):
-        raise NotImplementedError("BeautifulSoup objects don't support insert_before().")
-
-    def insert_after(self, successor):
-        raise NotImplementedError("BeautifulSoup objects don't support insert_after().")
-
-    def popTag(self):
-        tag = self.tagStack.pop()
-        #print "Pop", tag.name
-        if self.tagStack:
-            self.currentTag = self.tagStack[-1]
-        return self.currentTag
-
-    def pushTag(self, tag):
-        #print "Push", tag.name
-        if self.currentTag:
-            self.currentTag.contents.append(tag)
-        self.tagStack.append(tag)
-        self.currentTag = self.tagStack[-1]
-
-    def endData(self, containerClass=NavigableString):
-        if self.currentData:
-            currentData = u''.join(self.currentData)
-            if (currentData.translate(self.STRIP_ASCII_SPACES) == '' and
-                not set([tag.name for tag in self.tagStack]).intersection(
-                    self.builder.preserve_whitespace_tags)):
-                if '\n' in currentData:
-                    currentData = '\n'
-                else:
-                    currentData = ' '
-            self.currentData = []
-            if self.parse_only and len(self.tagStack) <= 1 and \
-                   (not self.parse_only.text or \
-                    not self.parse_only.search(currentData)):
-                return
-            o = containerClass(currentData)
-            self.object_was_parsed(o)
-
-    def object_was_parsed(self, o):
-        """Add an object to the parse tree."""
-        o.setup(self.currentTag, self.previous_element)
-        if self.previous_element:
-            self.previous_element.next_element = o
-        self.previous_element = o
-        self.currentTag.contents.append(o)
-
-    def _popToTag(self, name, nsprefix=None, inclusivePop=True):
-        """Pops the tag stack up to and including the most recent
-        instance of the given tag. If inclusivePop is false, pops the tag
-        stack up to but *not* including the most recent instqance of
-        the given tag."""
-        #print "Popping to %s" % name
-        if name == self.ROOT_TAG_NAME:
-            return
-
-        numPops = 0
-        mostRecentTag = None
-
-        for i in range(len(self.tagStack) - 1, 0, -1):
-            if (name == self.tagStack[i].name
-                and nsprefix == self.tagStack[i].prefix):
-                numPops = len(self.tagStack) - i
-                break
-        if not inclusivePop:
-            numPops = numPops - 1
-
-        for i in range(0, numPops):
-            mostRecentTag = self.popTag()
-        return mostRecentTag
-
-    def handle_starttag(self, name, namespace, nsprefix, attrs):
-        """Push a start tag on to the stack.
-
-        If this method returns None, the tag was rejected by the
-        SoupStrainer. You should proceed as if the tag had not occured
-        in the document. For instance, if this was a self-closing tag,
-        don't call handle_endtag.
-        """
-
-        # print "Start tag %s: %s" % (name, attrs)
-        self.endData()
-
-        if (self.parse_only and len(self.tagStack) <= 1
-            and (self.parse_only.text
-                 or not self.parse_only.search_tag(name, attrs))):
-            return None
-
-        tag = Tag(self, self.builder, name, namespace, nsprefix, attrs,
-                  self.currentTag, self.previous_element)
-        if tag is None:
-            return tag
-        if self.previous_element:
-            self.previous_element.next_element = tag
-        self.previous_element = tag
-        self.pushTag(tag)
-        return tag
-
-    def handle_endtag(self, name, nsprefix=None):
-        #print "End tag: " + name
-        self.endData()
-        self._popToTag(name, nsprefix)
-
-    def handle_data(self, data):
-        self.currentData.append(data)
-
-    def decode(self, pretty_print=False,
-               eventual_encoding=DEFAULT_OUTPUT_ENCODING,
-               formatter="minimal"):
-        """Returns a string or Unicode representation of this document.
-        To get Unicode, pass None for encoding."""
-
-        if self.is_xml:
-            # Print the XML declaration
-            encoding_part = ''
-            if eventual_encoding != None:
-                encoding_part = ' encoding="%s"' % eventual_encoding
-            prefix = u'<?xml version="1.0"%s?>\n' % encoding_part
-        else:
-            prefix = u''
-        if not pretty_print:
-            indent_level = None
-        else:
-            indent_level = 0
-        return prefix + super(BeautifulSoup, self).decode(
-            indent_level, eventual_encoding, formatter)
-
-class BeautifulStoneSoup(BeautifulSoup):
-    """Deprecated interface to an XML parser."""
-
-    def __init__(self, *args, **kwargs):
-        kwargs['features'] = 'xml'
-        warnings.warn(
-            'The BeautifulStoneSoup class is deprecated. Instead of using '
-            'it, pass features="xml" into the BeautifulSoup constructor.')
-        super(BeautifulStoneSoup, self).__init__(*args, **kwargs)
-
-
-class StopParsing(Exception):
-    pass
-
-
-class FeatureNotFound(ValueError):
-    pass
-
-
-#By default, act as an HTML pretty-printer.
-if __name__ == '__main__':
-    import sys
-    soup = BeautifulSoup(sys.stdin)
-    print soup.prettify()

bs4/builder/__init__.py

@@ -1,316 +0,0 @@
-from collections import defaultdict
-import itertools
-import sys
-from bs4.element import (
-    CharsetMetaAttributeValue,
-    ContentMetaAttributeValue,
-    whitespace_re
-    )
-
-__all__ = [
-    'HTMLTreeBuilder',
-    'SAXTreeBuilder',
-    'TreeBuilder',
-    'TreeBuilderRegistry',
-    ]
-
-# Some useful features for a TreeBuilder to have.
-FAST = 'fast'
-PERMISSIVE = 'permissive'
-STRICT = 'strict'
-XML = 'xml'
-HTML = 'html'
-HTML_5 = 'html5'
-
-
-class TreeBuilderRegistry(object):
-
-    def __init__(self):
-        self.builders_for_feature = defaultdict(list)
-        self.builders = []
-
-    def register(self, treebuilder_class):
-        """Register a treebuilder based on its advertised features."""
-        for feature in treebuilder_class.features:
-            self.builders_for_feature[feature].insert(0, treebuilder_class)
-        self.builders.insert(0, treebuilder_class)
-
-    def lookup(self, *features):
-        if len(self.builders) == 0:
-            # There are no builders at all.
-            return None
-
-        if len(features) == 0:
-            # They didn't ask for any features. Give them the most
-            # recently registered builder.
-            return self.builders[0]
-
-        # Go down the list of features in order, and eliminate any builders
-        # that don't match every feature.
-        features = list(features)
-        features.reverse()
-        candidates = None
-        candidate_set = None
-        while len(features) > 0:
-            feature = features.pop()
-            we_have_the_feature = self.builders_for_feature.get(feature, [])
-            if len(we_have_the_feature) > 0:
-                if candidates is None:
-                    candidates = we_have_the_feature
-                    candidate_set = set(candidates)
-                else:
-                    # Eliminate any candidates that don't have this feature.
-                    candidate_set = candidate_set.intersection(
-                        set(we_have_the_feature))
-
-        # The only valid candidates are the ones in candidate_set.
-        # Go through the original list of candidates and pick the first one
-        # that's in candidate_set.
-        if candidate_set is None:
-            return None
-        for candidate in candidates:
-            if candidate in candidate_set:
-                return candidate
-        return None
-
-# The BeautifulSoup class will take feature lists from developers and use them
-# to look up builders in this registry.
-builder_registry = TreeBuilderRegistry()
-
-class TreeBuilder(object):
-    """Turn a document into a Beautiful Soup object tree."""
-
-    features = []
-
-    is_xml = False
-    preserve_whitespace_tags = set()
-    empty_element_tags = None # A tag will be considered an empty-element
-                              # tag when and only when it has no contents.
-
-    # A value for these tag/attribute combinations is a space- or
-    # comma-separated list of CDATA, rather than a single CDATA.
-    cdata_list_attributes = {}
-
-
-    def __init__(self):
-        self.soup = None
-
-    def reset(self):
-        pass
-
-    def can_be_empty_element(self, tag_name):
-        """Might a tag with this name be an empty-element tag?
-
-        The final markup may or may not actually present this tag as
-        self-closing.
-
-        For instance: an HTMLBuilder does not consider a <p> tag to be
-        an empty-element tag (it's not in
-        HTMLBuilder.empty_element_tags). This means an empty <p> tag
-        will be presented as "<p></p>", not "<p />".
-
-        The default implementation has no opinion about which tags are
-        empty-element tags, so a tag will be presented as an
-        empty-element tag if and only if it has no contents.
-        "<foo></foo>" will become "<foo />", and "<foo>bar</foo>" will
-        be left alone.
-        """
-        if self.empty_element_tags is None:
-            return True
-        return tag_name in self.empty_element_tags
-
-    def feed(self, markup):
-        raise NotImplementedError()
-
-    def prepare_markup(self, markup, user_specified_encoding=None,
-                       document_declared_encoding=None):
-        return markup, None, None, False
-
-    def test_fragment_to_document(self, fragment):
-        """Wrap an HTML fragment to make it look like a document.
-
-        Different parsers do this differently. For instance, lxml
-        introduces an empty <head> tag, and html5lib
-        doesn't. Abstracting this away lets us write simple tests
-        which run HTML fragments through the parser and compare the
-        results against other HTML fragments.
-
-        This method should not be used outside of tests.
-        """
-        return fragment
-
-    def set_up_substitutions(self, tag):
-        return False
-
-    def _replace_cdata_list_attribute_values(self, tag_name, attrs):
-        """Replaces class="foo bar" with class=["foo", "bar"]
-
-        Modifies its input in place.
-        """
-        if self.cdata_list_attributes:
-            universal = self.cdata_list_attributes.get('*', [])
-            tag_specific = self.cdata_list_attributes.get(
-                tag_name.lower(), [])
-            for cdata_list_attr in itertools.chain(universal, tag_specific):
-                if cdata_list_attr in dict(attrs):
-                    # Basically, we have a "class" attribute whose
-                    # value is a whitespace-separated list of CSS
-                    # classes. Split it into a list.
-                    value = attrs[cdata_list_attr]
-                    if isinstance(value, basestring):
-                        values = whitespace_re.split(value)
-                    else:
-                        # html5lib sometimes calls setAttributes twice
-                        # for the same tag when rearranging the parse
-                        # tree. On the second call the attribute value
-                        # here is already a list.  If this happens,
-                        # leave the value alone rather than trying to
-                        # split it again.
-                        values = value
-                    attrs[cdata_list_attr] = values
-        return attrs
-
-class SAXTreeBuilder(TreeBuilder):
-    """A Beautiful Soup treebuilder that listens for SAX events."""
-
-    def feed(self, markup):
-        raise NotImplementedError()
-
-    def close(self):
-        pass
-
-    def startElement(self, name, attrs):
-        attrs = dict((key[1], value) for key, value in list(attrs.items()))
-        #print "Start %s, %r" % (name, attrs)
-        self.soup.handle_starttag(name, attrs)
-
-    def endElement(self, name):
-        #print "End %s" % name
-        self.soup.handle_endtag(name)
-
-    def startElementNS(self, nsTuple, nodeName, attrs):
-        # Throw away (ns, nodeName) for now.
-        self.startElement(nodeName, attrs)
-
-    def endElementNS(self, nsTuple, nodeName):
-        # Throw away (ns, nodeName) for now.
-        self.endElement(nodeName)
-        #handler.endElementNS((ns, node.nodeName), node.nodeName)
-
-    def startPrefixMapping(self, prefix, nodeValue):
-        # Ignore the prefix for now.
-        pass
-
-    def endPrefixMapping(self, prefix):
-        # Ignore the prefix for now.
-        # handler.endPrefixMapping(prefix)
-        pass
-
-    def characters(self, content):
-        self.soup.handle_data(content)
-
-    def startDocument(self):
-        pass
-
-    def endDocument(self):
-        pass
-
-
-class HTMLTreeBuilder(TreeBuilder):
-    """This TreeBuilder knows facts about HTML.
-
-    Such as which tags are empty-element tags.
-    """
-
-    preserve_whitespace_tags = set(['pre', 'textarea'])
-    empty_element_tags = set(['br' , 'hr', 'input', 'img', 'meta',
-                              'spacer', 'link', 'frame', 'base'])
-
-    # The HTML standard defines these attributes as containing a
-    # space-separated list of values, not a single value. That is,
-    # class="foo bar" means that the 'class' attribute has two values,
-    # 'foo' and 'bar', not the single value 'foo bar'.  When we
-    # encounter one of these attributes, we will parse its value into
-    # a list of values if possible. Upon output, the list will be
-    # converted back into a string.
-    cdata_list_attributes = {
-        "*" : ['class', 'accesskey', 'dropzone'],
-        "a" : ['rel', 'rev'],
-        "link" :  ['rel', 'rev'],
-        "td" : ["headers"],
-        "th" : ["headers"],
-        "td" : ["headers"],
-        "form" : ["accept-charset"],
-        "object" : ["archive"],
-
-        # These are HTML5 specific, as are *.accesskey and *.dropzone above.
-        "area" : ["rel"],
-        "icon" : ["sizes"],
-        "iframe" : ["sandbox"],
-        "output" : ["for"],
-        }
-
-    def set_up_substitutions(self, tag):
-        # We are only interested in <meta> tags
-        if tag.name != 'meta':
-            return False
-
-        http_equiv = tag.get('http-equiv')
-        content = tag.get('content')
-        charset = tag.get('charset')
-
-        # We are interested in <meta> tags that say what encoding the
-        # document was originally in. This means HTML 5-style <meta>
-        # tags that provide the "charset" attribute. It also means
-        # HTML 4-style <meta> tags that provide the "content"
-        # attribute and have "http-equiv" set to "content-type".
-        #
-        # In both cases we will replace the value of the appropriate
-        # attribute with a standin object that can take on any
-        # encoding.
-        meta_encoding = None
-        if charset is not None:
-            # HTML 5 style:
-            # <meta charset="utf8">
-            meta_encoding = charset
-            tag['charset'] = CharsetMetaAttributeValue(charset)
-
-        elif (content is not None and http_equiv is not None
-              and http_equiv.lower() == 'content-type'):
-            # HTML 4 style:
-            # <meta http-equiv="content-type" content="text/html; charset=utf8">
-            tag['content'] = ContentMetaAttributeValue(content)
-
-        return (meta_encoding is not None)
-
-def register_treebuilders_from(module):
-    """Copy TreeBuilders from the given module into this module."""
-    # I'm fairly sure this is not the best way to do this.
-    this_module = sys.modules['bs4.builder']
-    for name in module.__all__:
-        obj = getattr(module, name)
-
-        if issubclass(obj, TreeBuilder):
-            setattr(this_module, name, obj)
-            this_module.__all__.append(name)
-            # Register the builder while we're at it.
-            this_module.builder_registry.register(obj)
-
-# Builders are registered in reverse order of priority, so that custom
-# builder registrations will take precedence. In general, we want lxml
-# to take precedence over html5lib, because it's faster. And we only
-# want to use HTMLParser as a last result.
-from . import _htmlparser
-register_treebuilders_from(_htmlparser)
-try:
-    from . import _html5lib
-    register_treebuilders_from(_html5lib)
-except ImportError:
-    # They don't have html5lib installed.
-    pass
-try:
-    from . import _lxml
-    register_treebuilders_from(_lxml)
-except ImportError:
-    # They don't have lxml installed.
-    pass

bs4/builder/_html5lib.py

@@ -1,222 +0,0 @@
-__all__ = [
-    'HTML5TreeBuilder',
-    ]
-
-import warnings
-from bs4.builder import (
-    PERMISSIVE,
-    HTML,
-    HTML_5,
-    HTMLTreeBuilder,
-    )
-from bs4.element import NamespacedAttribute
-import html5lib
-from html5lib.constants import namespaces
-from bs4.element import (
-    Comment,
-    Doctype,
-    NavigableString,
-    Tag,
-    )
-
-class HTML5TreeBuilder(HTMLTreeBuilder):
-    """Use html5lib to build a tree."""
-
-    features = ['html5lib', PERMISSIVE, HTML_5, HTML]
-
-    def prepare_markup(self, markup, user_specified_encoding):
-        # Store the user-specified encoding for use later on.
-        self.user_specified_encoding = user_specified_encoding
-        return markup, None, None, False
-
-    # These methods are defined by Beautiful Soup.
-    def feed(self, markup):
-        if self.soup.parse_only is not None:
-            warnings.warn("You provided a value for parse_only, but the html5lib tree builder doesn't support parse_only. The entire document will be parsed.")
-        parser = html5lib.HTMLParser(tree=self.create_treebuilder)
-        doc = parser.parse(markup, encoding=self.user_specified_encoding)
-
-        # Set the character encoding detected by the tokenizer.
-        if isinstance(markup, unicode):
-            # We need to special-case this because html5lib sets
-            # charEncoding to UTF-8 if it gets Unicode input.
-            doc.original_encoding = None
-        else:
-            doc.original_encoding = parser.tokenizer.stream.charEncoding[0]
-
-    def create_treebuilder(self, namespaceHTMLElements):
-        self.underlying_builder = TreeBuilderForHtml5lib(
-            self.soup, namespaceHTMLElements)
-        return self.underlying_builder
-
-    def test_fragment_to_document(self, fragment):
-        """See `TreeBuilder`."""
-        return u'<html><head></head><body>%s</body></html>' % fragment
-
-
-class TreeBuilderForHtml5lib(html5lib.treebuilders._base.TreeBuilder):
-
-    def __init__(self, soup, namespaceHTMLElements):
-        self.soup = soup
-        super(TreeBuilderForHtml5lib, self).__init__(namespaceHTMLElements)
-
-    def documentClass(self):
-        self.soup.reset()
-        return Element(self.soup, self.soup, None)
-
-    def insertDoctype(self, token):
-        name = token["name"]
-        publicId = token["publicId"]
-        systemId = token["systemId"]
-
-        doctype = Doctype.for_name_and_ids(name, publicId, systemId)
-        self.soup.object_was_parsed(doctype)
-
-    def elementClass(self, name, namespace):
-        tag = self.soup.new_tag(name, namespace)
-        return Element(tag, self.soup, namespace)
-
-    def commentClass(self, data):
-        return TextNode(Comment(data), self.soup)
-
-    def fragmentClass(self):
-        self.soup = BeautifulSoup("")
-        self.soup.name = "[document_fragment]"
-        return Element(self.soup, self.soup, None)
-
-    def appendChild(self, node):
-        # XXX This code is not covered by the BS4 tests.
-        self.soup.append(node.element)
-
-    def getDocument(self):
-        return self.soup
-
-    def getFragment(self):
-        return html5lib.treebuilders._base.TreeBuilder.getFragment(self).element
-
-class AttrList(object):
-    def __init__(self, element):
-        self.element = element
-        self.attrs = dict(self.element.attrs)
-    def __iter__(self):
-        return list(self.attrs.items()).__iter__()
-    def __setitem__(self, name, value):
-        "set attr", name, value
-        self.element[name] = value
-    def items(self):
-        return list(self.attrs.items())
-    def keys(self):
-        return list(self.attrs.keys())
-    def __len__(self):
-        return len(self.attrs)
-    def __getitem__(self, name):
-        return self.attrs[name]
-    def __contains__(self, name):
-        return name in list(self.attrs.keys())
-
-
-class Element(html5lib.treebuilders._base.Node):
-    def __init__(self, element, soup, namespace):
-        html5lib.treebuilders._base.Node.__init__(self, element.name)
-        self.element = element
-        self.soup = soup
-        self.namespace = namespace
-
-    def appendChild(self, node):
-        if (node.element.__class__ == NavigableString and self.element.contents
-            and self.element.contents[-1].__class__ == NavigableString):
-            # Concatenate new text onto old text node
-            # XXX This has O(n^2) performance, for input like
-            # "a</a>a</a>a</a>..."
-            old_element = self.element.contents[-1]
-            new_element = self.soup.new_string(old_element + node.element)
-            old_element.replace_with(new_element)
-        else:
-            self.element.append(node.element)
-            node.parent = self
-
-    def getAttributes(self):
-        return AttrList(self.element)
-
-    def setAttributes(self, attributes):
-        if attributes is not None and len(attributes) > 0:
-
-            converted_attributes = []
-            for name, value in list(attributes.items()):
-                if isinstance(name, tuple):
-                    new_name = NamespacedAttribute(*name)
-                    del attributes[name]
-                    attributes[new_name] = value
-
-            self.soup.builder._replace_cdata_list_attribute_values(
-                self.name, attributes)
-            for name, value in attributes.items():
-                self.element[name] = value
-
-            # The attributes may contain variables that need substitution.
-            # Call set_up_substitutions manually.
-            #
-            # The Tag constructor called this method when the Tag was created,
-            # but we just set/changed the attributes, so call it again.
-            self.soup.builder.set_up_substitutions(self.element)
-    attributes = property(getAttributes, setAttributes)
-
-    def insertText(self, data, insertBefore=None):
-        text = TextNode(self.soup.new_string(data), self.soup)
-        if insertBefore:
-            self.insertBefore(text, insertBefore)
-        else:
-            self.appendChild(text)
-
-    def insertBefore(self, node, refNode):
-        index = self.element.index(refNode.element)
-        if (node.element.__class__ == NavigableString and self.element.contents
-            and self.element.contents[index-1].__class__ == NavigableString):
-            # (See comments in appendChild)
-            old_node = self.element.contents[index-1]
-            new_str = self.soup.new_string(old_node + node.element)
-            old_node.replace_with(new_str)
-        else:
-            self.element.insert(index, node.element)
-            node.parent = self
-
-    def removeChild(self, node):
-        node.element.extract()
-
-    def reparentChildren(self, newParent):
-        while self.element.contents:
-            child = self.element.contents[0]
-            child.extract()
-            if isinstance(child, Tag):
-                newParent.appendChild(
-                    Element(child, self.soup, namespaces["html"]))
-            else:
-                newParent.appendChild(
-                    TextNode(child, self.soup))
-
-    def cloneNode(self):
-        tag = self.soup.new_tag(self.element.name, self.namespace)
-        node = Element(tag, self.soup, self.namespace)
-        for key,value in self.attributes:
-            node.attributes[key] = value
-        return node
-
-    def hasContent(self):
-        return self.element.contents
-
-    def getNameTuple(self):
-        if self.namespace == None:
-            return namespaces["html"], self.name
-        else:
-            return self.namespace, self.name
-
-    nameTuple = property(getNameTuple)
-
-class TextNode(Element):
-    def __init__(self, element, soup):
-        html5lib.treebuilders._base.Node.__init__(self, None)
-        self.element = element
-        self.soup = soup
-
-    def cloneNode(self):
-        raise NotImplementedError

bs4/builder/_htmlparser.py

@@ -1,244 +0,0 @@
-"""Use the HTMLParser library to parse HTML files that aren't too bad."""
-
-__all__ = [
-    'HTMLParserTreeBuilder',
-    ]
-
-from HTMLParser import (
-    HTMLParser,
-    HTMLParseError,
-    )
-import sys
-import warnings
-
-# Starting in Python 3.2, the HTMLParser constructor takes a 'strict'
-# argument, which we'd like to set to False. Unfortunately,
-# http://bugs.python.org/issue13273 makes strict=True a better bet
-# before Python 3.2.3.
-#
-# At the end of this file, we monkeypatch HTMLParser so that
-# strict=True works well on Python 3.2.2.
-major, minor, release = sys.version_info[:3]
-CONSTRUCTOR_TAKES_STRICT = (
-    major > 3
-    or (major == 3 and minor > 2)
-    or (major == 3 and minor == 2 and release >= 3))
-
-from bs4.element import (
-    CData,
-    Comment,
-    Declaration,
-    Doctype,
-    ProcessingInstruction,
-    )
-from bs4.dammit import EntitySubstitution, UnicodeDammit
-
-from bs4.builder import (
-    HTML,
-    HTMLTreeBuilder,
-    STRICT,
-    )
-
-
-HTMLPARSER = 'html.parser'
-
-class BeautifulSoupHTMLParser(HTMLParser):
-    def handle_starttag(self, name, attrs):
-        # XXX namespace
-        self.soup.handle_starttag(name, None, None, dict(attrs))
-
-    def handle_endtag(self, name):
-        self.soup.handle_endtag(name)
-
-    def handle_data(self, data):
-        self.soup.handle_data(data)
-
-    def handle_charref(self, name):
-        # XXX workaround for a bug in HTMLParser. Remove this once
-        # it's fixed.
-        if name.startswith('x'):
-            real_name = int(name.lstrip('x'), 16)
-        else:
-            real_name = int(name)
-
-        try:
-            data = unichr(real_name)
-        except (ValueError, OverflowError), e:
-            data = u"\N{REPLACEMENT CHARACTER}"
-
-        self.handle_data(data)
-
-    def handle_entityref(self, name):
-        character = EntitySubstitution.HTML_ENTITY_TO_CHARACTER.get(name)
-        if character is not None:
-            data = character
-        else:
-            data = "&%s;" % name
-        self.handle_data(data)
-
-    def handle_comment(self, data):
-        self.soup.endData()
-        self.soup.handle_data(data)
-        self.soup.endData(Comment)
-
-    def handle_decl(self, data):
-        self.soup.endData()
-        if data.startswith("DOCTYPE "):
-            data = data[len("DOCTYPE "):]
-        self.soup.handle_data(data)
-        self.soup.endData(Doctype)
-
-    def unknown_decl(self, data):
-        if data.upper().startswith('CDATA['):
-            cls = CData
-            data = data[len('CDATA['):]
-        else:
-            cls = Declaration
-        self.soup.endData()
-        self.soup.handle_data(data)
-        self.soup.endData(cls)
-
-    def handle_pi(self, data):
-        self.soup.endData()
-        if data.endswith("?") and data.lower().startswith("xml"):
-            # "An XHTML processing instruction using the trailing '?'
-            # will cause the '?' to be included in data." - HTMLParser
-            # docs.
-            #
-            # Strip the question mark so we don't end up with two
-            # question marks.
-            data = data[:-1]
-        self.soup.handle_data(data)
-        self.soup.endData(ProcessingInstruction)
-
-
-class HTMLParserTreeBuilder(HTMLTreeBuilder):
-
-    is_xml = False
-    features = [HTML, STRICT, HTMLPARSER]
-
-    def __init__(self, *args, **kwargs):
-        if CONSTRUCTOR_TAKES_STRICT:
-            kwargs['strict'] = False
-        self.parser_args = (args, kwargs)
-
-    def prepare_markup(self, markup, user_specified_encoding=None,
-                       document_declared_encoding=None):
-        """
-        :return: A 4-tuple (markup, original encoding, encoding
-        declared within markup, whether any characters had to be
-        replaced with REPLACEMENT CHARACTER).
-        """
-        if isinstance(markup, unicode):
-            return markup, None, None, False
-
-        try_encodings = [user_specified_encoding, document_declared_encoding]
-        dammit = UnicodeDammit(markup, try_encodings, is_html=True)
-        return (dammit.markup, dammit.original_encoding,
-                dammit.declared_html_encoding,
-                dammit.contains_replacement_characters)
-
-    def feed(self, markup):
-        args, kwargs = self.parser_args
-        parser = BeautifulSoupHTMLParser(*args, **kwargs)
-        parser.soup = self.soup
-        try:
-            parser.feed(markup)
-        except HTMLParseError, e:
-            warnings.warn(RuntimeWarning(
-                "Python's built-in HTMLParser cannot parse the given document. This is not a bug in Beautiful Soup. The best solution is to install an external parser (lxml or html5lib), and use Beautiful Soup with that parser. See http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser for help."))
-            raise e
-
-# Patch 3.2 versions of HTMLParser earlier than 3.2.3 to use some
-# 3.2.3 code. This ensures they don't treat markup like <p></p> as a
-# string.
-#
-# XXX This code can be removed once most Python 3 users are on 3.2.3.
-if major == 3 and minor == 2 and not CONSTRUCTOR_TAKES_STRICT:
-    import re
-    attrfind_tolerant = re.compile(
-        r'\s*((?<=[\'"\s])[^\s/>][^\s/=>]*)(\s*=+\s*'
-        r'(\'[^\']*\'|"[^"]*"|(?![\'"])[^>\s]*))?')
-    HTMLParserTreeBuilder.attrfind_tolerant = attrfind_tolerant
-
-    locatestarttagend = re.compile(r"""
-  <[a-zA-Z][-.a-zA-Z0-9:_]*          # tag name
-  (?:\s+                             # whitespace before attribute name
-    (?:[a-zA-Z_][-.:a-zA-Z0-9_]*     # attribute name
-      (?:\s*=\s*                     # value indicator
-        (?:'[^']*'                   # LITA-enclosed value
-          |\"[^\"]*\"                # LIT-enclosed value
-          |[^'\">\s]+                # bare value
-         )
-       )?
-     )
-   )*
-  \s*                                # trailing whitespace
-""", re.VERBOSE)
-    BeautifulSoupHTMLParser.locatestarttagend = locatestarttagend
-
-    from html.parser import tagfind, attrfind
-
-    def parse_starttag(self, i):
-        self.__starttag_text = None
-        endpos = self.check_for_whole_start_tag(i)
-        if endpos < 0:
-            return endpos
-        rawdata = self.rawdata
-        self.__starttag_text = rawdata[i:endpos]
-
-        # Now parse the data between i+1 and j into a tag and attrs
-        attrs = []
-        match = tagfind.match(rawdata, i+1)
-        assert match, 'unexpected call to parse_starttag()'
-        k = match.end()
-        self.lasttag = tag = rawdata[i+1:k].lower()
-        while k < endpos:
-            if self.strict:
-                m = attrfind.match(rawdata, k)
-            else:
-                m = attrfind_tolerant.match(rawdata, k)
-            if not m:
-                break
-            attrname, rest, attrvalue = m.group(1, 2, 3)
-            if not rest:
-                attrvalue = None
-            elif attrvalue[:1] == '\'' == attrvalue[-1:] or \
-                 attrvalue[:1] == '"' == attrvalue[-1:]:
-                attrvalue = attrvalue[1:-1]
-            if attrvalue:
-                attrvalue = self.unescape(attrvalue)
-            attrs.append((attrname.lower(), attrvalue))
-            k = m.end()
-
-        end = rawdata[k:endpos].strip()
-        if end not in (">", "/>"):
-            lineno, offset = self.getpos()
-            if "\n" in self.__starttag_text:
-                lineno = lineno + self.__starttag_text.count("\n")
-                offset = len(self.__starttag_text) \
-                         - self.__starttag_text.rfind("\n")
-            else:
-                offset = offset + len(self.__starttag_text)
-            if self.strict:
-                self.error("junk characters in start tag: %r"
-                           % (rawdata[k:endpos][:20],))
-            self.handle_data(rawdata[i:endpos])
-            return endpos
-        if end.endswith('/>'):
-            # XHTML-style empty tag: <span attr="value" />
-            self.handle_startendtag(tag, attrs)
-        else:
-            self.handle_starttag(tag, attrs)
-            if tag in self.CDATA_CONTENT_ELEMENTS:
-                self.set_cdata_mode(tag)
-        return endpos
-
-    def set_cdata_mode(self, elem):
-        self.cdata_elem = elem.lower()
-        self.interesting = re.compile(r'</\s*%s\s*>' % self.cdata_elem, re.I)
-
-    BeautifulSoupHTMLParser.parse_starttag = parse_starttag
-    BeautifulSoupHTMLParser.set_cdata_mode = set_cdata_mode
-
-    CONSTRUCTOR_TAKES_STRICT = True

bs4/builder/_lxml.py

@@ -1,199 +0,0 @@
-__all__ = [
-    'LXMLTreeBuilderForXML',
-    'LXMLTreeBuilder',
-    ]
-
-from StringIO import StringIO
-import collections
-from lxml import etree
-from bs4.element import Comment, Doctype, NamespacedAttribute
-from bs4.builder import (
-    FAST,
-    HTML,
-    HTMLTreeBuilder,
-    PERMISSIVE,
-    TreeBuilder,
-    XML)
-from bs4.dammit import UnicodeDammit
-
-LXML = 'lxml'
-
-class LXMLTreeBuilderForXML(TreeBuilder):
-    DEFAULT_PARSER_CLASS = etree.XMLParser
-
-    is_xml = True
-
-    # Well, it's permissive by XML parser standards.
-    features = [LXML, XML, FAST, PERMISSIVE]
-
-    CHUNK_SIZE = 512
-
-    @property
-    def default_parser(self):
-        # This can either return a parser object or a class, which
-        # will be instantiated with default arguments.
-        return etree.XMLParser(target=self, strip_cdata=False, recover=True)
-
-    def __init__(self, parser=None, empty_element_tags=None):
-        if empty_element_tags is not None:
-            self.empty_element_tags = set(empty_element_tags)
-        if parser is None:
-            # Use the default parser.
-            parser = self.default_parser
-        if isinstance(parser, collections.Callable):
-            # Instantiate the parser with default arguments
-            parser = parser(target=self, strip_cdata=False)
-        self.parser = parser
-        self.soup = None
-        self.nsmaps = None
-
-    def _getNsTag(self, tag):
-        # Split the namespace URL out of a fully-qualified lxml tag
-        # name. Copied from lxml's src/lxml/sax.py.
-        if tag[0] == '{':
-            return tuple(tag[1:].split('}', 1))
-        else:
-            return (None, tag)
-
-    def prepare_markup(self, markup, user_specified_encoding=None,
-                       document_declared_encoding=None):
-        """
-        :return: A 3-tuple (markup, original encoding, encoding
-        declared within markup).
-        """
-        if isinstance(markup, unicode):
-            return markup, None, None, False
-
-        try_encodings = [user_specified_encoding, document_declared_encoding]
-        dammit = UnicodeDammit(markup, try_encodings, is_html=True)
-        return (dammit.markup, dammit.original_encoding,
-                dammit.declared_html_encoding,
-                dammit.contains_replacement_characters)
-
-    def feed(self, markup):
-        if isinstance(markup, basestring):
-            markup = StringIO(markup)
-        # Call feed() at least once, even if the markup is empty,
-        # or the parser won't be initialized.
-        data = markup.read(self.CHUNK_SIZE)
-        self.parser.feed(data)
-        while data != '':
-            # Now call feed() on the rest of the data, chunk by chunk.
-            data = markup.read(self.CHUNK_SIZE)
-            if data != '':
-                self.parser.feed(data)
-        self.parser.close()
-
-    def close(self):
-        self.nsmaps = None
-
-    def start(self, name, attrs, nsmap={}):
-        # Make sure attrs is a mutable dict--lxml may send an immutable dictproxy.
-        attrs = dict(attrs)
-
-        nsprefix = None
-        # Invert each namespace map as it comes in.
-        if len(nsmap) == 0 and self.nsmaps != None:
-            # There are no new namespaces for this tag, but namespaces
-            # are in play, so we need a separate tag stack to know
-            # when they end.
-            self.nsmaps.append(None)
-        elif len(nsmap) > 0:
-            # A new namespace mapping has come into play.
-            if self.nsmaps is None:
-                self.nsmaps = []
-            inverted_nsmap = dict((value, key) for key, value in nsmap.items())
-            self.nsmaps.append(inverted_nsmap)
-            # Also treat the namespace mapping as a set of attributes on the
-            # tag, so we can recreate it later.
-            attrs = attrs.copy()
-            for prefix, namespace in nsmap.items():
-                attribute = NamespacedAttribute(
-                    "xmlns", prefix, "http://www.w3.org/2000/xmlns/")
-                attrs[attribute] = namespace
-
-        if self.nsmaps is not None and len(self.nsmaps) > 0:
-            # Namespaces are in play. Find any attributes that came in
-            # from lxml with namespaces attached to their names, and
-            # turn then into NamespacedAttribute objects.
-            new_attrs = {}
-            for attr, value in attrs.items():
-                namespace, attr = self._getNsTag(attr)
-                if namespace is None:
-                    new_attrs[attr] = value
-                else:
-                    nsprefix = self._prefix_for_namespace(namespace)
-                    attr = NamespacedAttribute(nsprefix, attr, namespace)
-                    new_attrs[attr] = value
-            attrs = new_attrs
-
-        namespace, name = self._getNsTag(name)
-        nsprefix = self._prefix_for_namespace(namespace)
-        self.soup.handle_starttag(name, namespace, nsprefix, attrs)
-
-    def _prefix_for_namespace(self, namespace):
-        """Find the currently active prefix for the given namespace."""
-        if namespace is None:
-            return None
-        for inverted_nsmap in reversed(self.nsmaps):
-            if inverted_nsmap is not None and namespace in inverted_nsmap:
-                return inverted_nsmap[namespace]
-
-    def end(self, name):
-        self.soup.endData()
-        completed_tag = self.soup.tagStack[-1]
-        namespace, name = self._getNsTag(name)
-        nsprefix = None
-        if namespace is not None:
-            for inverted_nsmap in reversed(self.nsmaps):
-                if inverted_nsmap is not None and namespace in inverted_nsmap:
-                    nsprefix = inverted_nsmap[namespace]
-                    break
-        self.soup.handle_endtag(name, nsprefix)
-        if self.nsmaps != None:
-            # This tag, or one of its parents, introduced a namespace
-            # mapping, so pop it off the stack.
-            self.nsmaps.pop()
-            if len(self.nsmaps) == 0:
-                # Namespaces are no longer in play, so don't bother keeping
-                # track of the namespace stack.
-                self.nsmaps = None
-
-    def pi(self, target, data):
-        pass
-
-    def data(self, content):
-        self.soup.handle_data(content)
-
-    def doctype(self, name, pubid, system):
-        self.soup.endData()
-        doctype = Doctype.for_name_and_ids(name, pubid, system)
-        self.soup.object_was_parsed(doctype)
-
-    def comment(self, content):
-        "Handle comments as Comment objects."
-        self.soup.endData()
-        self.soup.handle_data(content)
-        self.soup.endData(Comment)
-
-    def test_fragment_to_document(self, fragment):
-        """See `TreeBuilder`."""
-        return u'<?xml version="1.0" encoding="utf-8"?>\n%s' % fragment
-
-
-class LXMLTreeBuilder(HTMLTreeBuilder, LXMLTreeBuilderForXML):
-
-    features = [LXML, HTML, FAST, PERMISSIVE]
-    is_xml = False
-
-    @property
-    def default_parser(self):
-        return etree.HTMLParser
-
-    def feed(self, markup):
-        self.parser.feed(markup)
-        self.parser.close()
-
-    def test_fragment_to_document(self, fragment):
-        """See `TreeBuilder`."""
-        return u'<html><body>%s</body></html>' % fragment

bs4/dammit.py

@@ -1,803 +0,0 @@
-# -*- coding: utf-8 -*-
-"""Beautiful Soup bonus library: Unicode, Dammit
-
-This class forces XML data into a standard format (usually to UTF-8 or
-Unicode).  It is heavily based on code from Mark Pilgrim's Universal
-Feed Parser. It does not rewrite the XML or HTML to reflect a new
-encoding; that's the tree builder's job.
-"""
-
-import codecs
-from htmlentitydefs import codepoint2name
-import re
-import logging
-
-# Import a library to autodetect character encodings.
-chardet_type = None
-try:
-    # First try the fast C implementation.
-    #  PyPI package: cchardet
-    import cchardet
-    def chardet_dammit(s):
-        return cchardet.detect(s)['encoding']
-except ImportError:
-    try:
-        # Fall back to the pure Python implementation
-        #  Debian package: python-chardet
-        #  PyPI package: chardet
-        import chardet
-        def chardet_dammit(s):
-            return chardet.detect(s)['encoding']
-        #import chardet.constants
-        #chardet.constants._debug = 1
-    except ImportError:
-        # No chardet available.
-        def chardet_dammit(s):
-            return None
-
-# Available from http://cjkpython.i18n.org/.
-try:
-    import iconv_codec
-except ImportError:
-    pass
-
-xml_encoding_re = re.compile(
-    '^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
-html_meta_re = re.compile(
-    '<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(), re.I)
-
-class EntitySubstitution(object):
-
-    """Substitute XML or HTML entities for the corresponding characters."""
-
-    def _populate_class_variables():
-        lookup = {}
-        reverse_lookup = {}
-        characters_for_re = []
-        for codepoint, name in list(codepoint2name.items()):
-            character = unichr(codepoint)
-            if codepoint != 34:
-                # There's no point in turning the quotation mark into
-                # &quot;, unless it happens within an attribute value, which
-                # is handled elsewhere.
-                characters_for_re.append(character)
-                lookup[character] = name
-            # But we do want to turn &quot; into the quotation mark.
-            reverse_lookup[name] = character
-        re_definition = "[%s]" % "".join(characters_for_re)
-        return lookup, reverse_lookup, re.compile(re_definition)
-    (CHARACTER_TO_HTML_ENTITY, HTML_ENTITY_TO_CHARACTER,
-     CHARACTER_TO_HTML_ENTITY_RE) = _populate_class_variables()
-
-    CHARACTER_TO_XML_ENTITY = {
-        "'": "apos",
-        '"': "quot",
-        "&": "amp",
-        "<": "lt",
-        ">": "gt",
-        }
-
-    BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
-                                           "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
-                                           ")")
-
-    @classmethod
-    def _substitute_html_entity(cls, matchobj):
-        entity = cls.CHARACTER_TO_HTML_ENTITY.get(matchobj.group(0))
-        return "&%s;" % entity
-
-    @classmethod
-    def _substitute_xml_entity(cls, matchobj):
-        """Used with a regular expression to substitute the
-        appropriate XML entity for an XML special character."""
-        entity = cls.CHARACTER_TO_XML_ENTITY[matchobj.group(0)]
-        return "&%s;" % entity
-
-    @classmethod
-    def quoted_attribute_value(self, value):
-        """Make a value into a quoted XML attribute, possibly escaping it.
-
-         Most strings will be quoted using double quotes.
-
-          Bob's Bar -> "Bob's Bar"
-
-         If a string contains double quotes, it will be quoted using
-         single quotes.
-
-          Welcome to "my bar" -> 'Welcome to "my bar"'
-
-         If a string contains both single and double quotes, the
-         double quotes will be escaped, and the string will be quoted
-         using double quotes.
-
-          Welcome to "Bob's Bar" -> "Welcome to &quot;Bob's bar&quot;
-        """
-        quote_with = '"'
-        if '"' in value:
-            if "'" in value:
-                # The string contains both single and double
-                # quotes.  Turn the double quotes into
-                # entities. We quote the double quotes rather than
-                # the single quotes because the entity name is
-                # "&quot;" whether this is HTML or XML.  If we
-                # quoted the single quotes, we'd have to decide
-                # between &apos; and &squot;.
-                replace_with = "&quot;"
-                value = value.replace('"', replace_with)
-            else:
-                # There are double quotes but no single quotes.
-                # We can use single quotes to quote the attribute.
-                quote_with = "'"
-        return quote_with + value + quote_with
-
-    @classmethod
-    def substitute_xml(cls, value, make_quoted_attribute=False):
-        """Substitute XML entities for special XML characters.
-
-        :param value: A string to be substituted. The less-than sign will
-          become &lt;, the greater-than sign will become &gt;, and any
-          ampersands that are not part of an entity defition will
-          become &amp;.
-
-        :param make_quoted_attribute: If True, then the string will be
-         quoted, as befits an attribute value.
-        """
-        # Escape angle brackets, and ampersands that aren't part of
-        # entities.
-        value = cls.BARE_AMPERSAND_OR_BRACKET.sub(
-            cls._substitute_xml_entity, value)
-
-        if make_quoted_attribute:
-            value = cls.quoted_attribute_value(value)
-        return value
-
-    @classmethod
-    def substitute_html(cls, s):
-        """Replace certain Unicode characters with named HTML entities.
-
-        This differs from data.encode(encoding, 'xmlcharrefreplace')
-        in that the goal is to make the result more readable (to those
-        with ASCII displays) rather than to recover from
-        errors. There's absolutely nothing wrong with a UTF-8 string
-        containg a LATIN SMALL LETTER E WITH ACUTE, but replacing that
-        character with "&eacute;" will make it more readable to some
-        people.
-        """
-        return cls.CHARACTER_TO_HTML_ENTITY_RE.sub(
-            cls._substitute_html_entity, s)
-
-
-class UnicodeDammit:
-    """A class for detecting the encoding of a *ML document and
-    converting it to a Unicode string. If the source encoding is
-    windows-1252, can replace MS smart quotes with their HTML or XML
-    equivalents."""
-
-    # This dictionary maps commonly seen values for "charset" in HTML
-    # meta tags to the corresponding Python codec names. It only covers
-    # values that aren't in Python's aliases and can't be determined
-    # by the heuristics in find_codec.
-    CHARSET_ALIASES = {"macintosh": "mac-roman",
-                       "x-sjis": "shift-jis"}
-
-    ENCODINGS_WITH_SMART_QUOTES = [
-        "windows-1252",
-        "iso-8859-1",
-        "iso-8859-2",
-        ]
-
-    def __init__(self, markup, override_encodings=[],
-                 smart_quotes_to=None, is_html=False):
-        self.declared_html_encoding = None
-        self.smart_quotes_to = smart_quotes_to
-        self.tried_encodings = []
-        self.contains_replacement_characters = False
-
-        if markup == '' or isinstance(markup, unicode):
-            self.markup = markup
-            self.unicode_markup = unicode(markup)
-            self.original_encoding = None
-            return
-
-        new_markup, document_encoding, sniffed_encoding = \
-            self._detectEncoding(markup, is_html)
-        self.markup = new_markup
-
-        u = None
-        if new_markup != markup:
-            # _detectEncoding modified the markup, then converted it to
-            # Unicode and then to UTF-8. So convert it from UTF-8.
-            u = self._convert_from("utf8")
-            self.original_encoding = sniffed_encoding
-
-        if not u:
-            for proposed_encoding in (
-                override_encodings + [document_encoding, sniffed_encoding]):
-                if proposed_encoding is not None:
-                    u = self._convert_from(proposed_encoding)
-                    if u:
-                        break
-
-        # If no luck and we have auto-detection library, try that:
-        if not u and not isinstance(self.markup, unicode):
-            u = self._convert_from(chardet_dammit(self.markup))
-
-        # As a last resort, try utf-8 and windows-1252:
-        if not u:
-            for proposed_encoding in ("utf-8", "windows-1252"):
-                u = self._convert_from(proposed_encoding)
-                if u:
-                    break
-
-        # As an absolute last resort, try the encodings again with
-        # character replacement.
-        if not u:
-            for proposed_encoding in (
-                override_encodings + [
-                    document_encoding, sniffed_encoding, "utf-8", "windows-1252"]):
-                if proposed_encoding != "ascii":
-                    u = self._convert_from(proposed_encoding, "replace")
-                if u is not None:
-                    logging.warning(
-                            "Some characters could not be decoded, and were "
-                            "replaced with REPLACEMENT CHARACTER.")
-                    self.contains_replacement_characters = True
-                    break
-
-        # We could at this point force it to ASCII, but that would
-        # destroy so much data that I think giving up is better
-        self.unicode_markup = u
-        if not u:
-            self.original_encoding = None
-
-    def _sub_ms_char(self, match):
-        """Changes a MS smart quote character to an XML or HTML
-        entity, or an ASCII character."""
-        orig = match.group(1)
-        if self.smart_quotes_to == 'ascii':
-            sub = self.MS_CHARS_TO_ASCII.get(orig).encode()
-        else:
-            sub = self.MS_CHARS.get(orig)
-            if type(sub) == tuple:
-                if self.smart_quotes_to == 'xml':
-                    sub = '&#x'.encode() + sub[1].encode() + ';'.encode()
-                else:
-                    sub = '&'.encode() + sub[0].encode() + ';'.encode()
-            else:
-                sub = sub.encode()
-        return sub
-
-    def _convert_from(self, proposed, errors="strict"):
-        proposed = self.find_codec(proposed)
-        if not proposed or (proposed, errors) in self.tried_encodings:
-            return None
-        self.tried_encodings.append((proposed, errors))
-        markup = self.markup
-
-        # Convert smart quotes to HTML if coming from an encoding
-        # that might have them.
-        if (self.smart_quotes_to is not None
-            and proposed.lower() in self.ENCODINGS_WITH_SMART_QUOTES):
-            smart_quotes_re = b"([\x80-\x9f])"
-            smart_quotes_compiled = re.compile(smart_quotes_re)
-            markup = smart_quotes_compiled.sub(self._sub_ms_char, markup)
-
-        try:
-            #print "Trying to convert document to %s (errors=%s)" % (
-            #    proposed, errors)
-            u = self._to_unicode(markup, proposed, errors)
-            self.markup = u
-            self.original_encoding = proposed
-        except Exception as e:
-            #print "That didn't work!"
-            #print e
-            return None
-        #print "Correct encoding: %s" % proposed
-        return self.markup
-
-    def _to_unicode(self, data, encoding, errors="strict"):
-        '''Given a string and its encoding, decodes the string into Unicode.
-        %encoding is a string recognized by encodings.aliases'''
-
-        # strip Byte Order Mark (if present)
-        if (len(data) >= 4) and (data[:2] == '\xfe\xff') \
-               and (data[2:4] != '\x00\x00'):
-            encoding = 'utf-16be'
-            data = data[2:]
-        elif (len(data) >= 4) and (data[:2] == '\xff\xfe') \
-                 and (data[2:4] != '\x00\x00'):
-            encoding = 'utf-16le'
-            data = data[2:]
-        elif data[:3] == '\xef\xbb\xbf':
-            encoding = 'utf-8'
-            data = data[3:]
-        elif data[:4] == '\x00\x00\xfe\xff':
-            encoding = 'utf-32be'
-            data = data[4:]
-        elif data[:4] == '\xff\xfe\x00\x00':
-            encoding = 'utf-32le'
-            data = data[4:]
-        newdata = unicode(data, encoding, errors)
-        return newdata
-
-    def _detectEncoding(self, xml_data, is_html=False):
-        """Given a document, tries to detect its XML encoding."""
-        xml_encoding = sniffed_xml_encoding = None
-        try:
-            if xml_data[:4] == b'\x4c\x6f\xa7\x94':
-                # EBCDIC
-                xml_data = self._ebcdic_to_ascii(xml_data)
-            elif xml_data[:4] == b'\x00\x3c\x00\x3f':
-                # UTF-16BE
-                sniffed_xml_encoding = 'utf-16be'
-                xml_data = unicode(xml_data, 'utf-16be').encode('utf-8')
-            elif (len(xml_data) >= 4) and (xml_data[:2] == b'\xfe\xff') \
-                     and (xml_data[2:4] != b'\x00\x00'):
-                # UTF-16BE with BOM
-                sniffed_xml_encoding = 'utf-16be'
-                xml_data = unicode(xml_data[2:], 'utf-16be').encode('utf-8')
-            elif xml_data[:4] == b'\x3c\x00\x3f\x00':
-                # UTF-16LE
-                sniffed_xml_encoding = 'utf-16le'
-                xml_data = unicode(xml_data, 'utf-16le').encode('utf-8')
-            elif (len(xml_data) >= 4) and (xml_data[:2] == b'\xff\xfe') and \
-                     (xml_data[2:4] != b'\x00\x00'):
-                # UTF-16LE with BOM
-                sniffed_xml_encoding = 'utf-16le'
-                xml_data = unicode(xml_data[2:], 'utf-16le').encode('utf-8')
-            elif xml_data[:4] == b'\x00\x00\x00\x3c':
-                # UTF-32BE
-                sniffed_xml_encoding = 'utf-32be'
-                xml_data = unicode(xml_data, 'utf-32be').encode('utf-8')
-            elif xml_data[:4] == b'\x3c\x00\x00\x00':
-                # UTF-32LE
-                sniffed_xml_encoding = 'utf-32le'
-                xml_data = unicode(xml_data, 'utf-32le').encode('utf-8')
-            elif xml_data[:4] == b'\x00\x00\xfe\xff':
-                # UTF-32BE with BOM
-                sniffed_xml_encoding = 'utf-32be'
-                xml_data = unicode(xml_data[4:], 'utf-32be').encode('utf-8')
-            elif xml_data[:4] == b'\xff\xfe\x00\x00':
-                # UTF-32LE with BOM
-                sniffed_xml_encoding = 'utf-32le'
-                xml_data = unicode(xml_data[4:], 'utf-32le').encode('utf-8')
-            elif xml_data[:3] == b'\xef\xbb\xbf':
-                # UTF-8 with BOM
-                sniffed_xml_encoding = 'utf-8'
-                xml_data = unicode(xml_data[3:], 'utf-8').encode('utf-8')
-            else:
-                sniffed_xml_encoding = 'ascii'
-                pass
-        except:
-            xml_encoding_match = None
-        xml_encoding_match = xml_encoding_re.match(xml_data)
-        if not xml_encoding_match and is_html:
-            xml_encoding_match = html_meta_re.search(xml_data)
-        if xml_encoding_match is not None:
-            xml_encoding = xml_encoding_match.groups()[0].decode(
-                'ascii').lower()
-            if is_html:
-                self.declared_html_encoding = xml_encoding
-            if sniffed_xml_encoding and \
-               (xml_encoding in ('iso-10646-ucs-2', 'ucs-2', 'csunicode',
-                                 'iso-10646-ucs-4', 'ucs-4', 'csucs4',
-                                 'utf-16', 'utf-32', 'utf_16', 'utf_32',
-                                 'utf16', 'u16')):
-                xml_encoding = sniffed_xml_encoding
-        return xml_data, xml_encoding, sniffed_xml_encoding
-
-    def find_codec(self, charset):
-        return self._codec(self.CHARSET_ALIASES.get(charset, charset)) \
-               or (charset and self._codec(charset.replace("-", ""))) \
-               or (charset and self._codec(charset.replace("-", "_"))) \
-               or charset
-
-    def _codec(self, charset):
-        if not charset:
-            return charset
-        codec = None
-        try:
-            codecs.lookup(charset)
-            codec = charset
-        except (LookupError, ValueError):
-            pass
-        return codec
-
-    EBCDIC_TO_ASCII_MAP = None
-
-    def _ebcdic_to_ascii(self, s):
-        c = self.__class__
-        if not c.EBCDIC_TO_ASCII_MAP:
-            emap = (0,1,2,3,156,9,134,127,151,141,142,11,12,13,14,15,
-                    16,17,18,19,157,133,8,135,24,25,146,143,28,29,30,31,
-                    128,129,130,131,132,10,23,27,136,137,138,139,140,5,6,7,
-                    144,145,22,147,148,149,150,4,152,153,154,155,20,21,158,26,
-                    32,160,161,162,163,164,165,166,167,168,91,46,60,40,43,33,
-                    38,169,170,171,172,173,174,175,176,177,93,36,42,41,59,94,
-                    45,47,178,179,180,181,182,183,184,185,124,44,37,95,62,63,
-                    186,187,188,189,190,191,192,193,194,96,58,35,64,39,61,34,
-                    195,97,98,99,100,101,102,103,104,105,196,197,198,199,200,
-                    201,202,106,107,108,109,110,111,112,113,114,203,204,205,
-                    206,207,208,209,126,115,116,117,118,119,120,121,122,210,
-                    211,212,213,214,215,216,217,218,219,220,221,222,223,224,
-                    225,226,227,228,229,230,231,123,65,66,67,68,69,70,71,72,
-                    73,232,233,234,235,236,237,125,74,75,76,77,78,79,80,81,
-                    82,238,239,240,241,242,243,92,159,83,84,85,86,87,88,89,
-                    90,244,245,246,247,248,249,48,49,50,51,52,53,54,55,56,57,
-                    250,251,252,253,254,255)
-            import string
-            c.EBCDIC_TO_ASCII_MAP = string.maketrans(
-            ''.join(map(chr, list(range(256)))), ''.join(map(chr, emap)))
-        return s.translate(c.EBCDIC_TO_ASCII_MAP)
-
-    # A partial mapping of ISO-Latin-1 to HTML entities/XML numeric entities.
-    MS_CHARS = {b'\x80': ('euro', '20AC'),
-                b'\x81': ' ',
-                b'\x82': ('sbquo', '201A'),
-                b'\x83': ('fnof', '192'),
-                b'\x84': ('bdquo', '201E'),
-                b'\x85': ('hellip', '2026'),
-                b'\x86': ('dagger', '2020'),
-                b'\x87': ('Dagger', '2021'),
-                b'\x88': ('circ', '2C6'),
-                b'\x89': ('permil', '2030'),
-                b'\x8A': ('Scaron', '160'),
-                b'\x8B': ('lsaquo', '2039'),
-                b'\x8C': ('OElig', '152'),
-                b'\x8D': '?',
-                b'\x8E': ('#x17D', '17D'),
-                b'\x8F': '?',
-                b'\x90': '?',
-                b'\x91': ('lsquo', '2018'),
-                b'\x92': ('rsquo', '2019'),
-                b'\x93': ('ldquo', '201C'),
-                b'\x94': ('rdquo', '201D'),
-                b'\x95': ('bull', '2022'),
-                b'\x96': ('ndash', '2013'),
-                b'\x97': ('mdash', '2014'),
-                b'\x98': ('tilde', '2DC'),
-                b'\x99': ('trade', '2122'),
-                b'\x9a': ('scaron', '161'),
-                b'\x9b': ('rsaquo', '203A'),
-                b'\x9c': ('oelig', '153'),
-                b'\x9d': '?',
-                b'\x9e': ('#x17E', '17E'),
-                b'\x9f': ('Yuml', ''),}
-
-    # A parochial partial mapping of ISO-Latin-1 to ASCII. Contains
-    # horrors like stripping diacritical marks to turn á into a, but also
-    # contains non-horrors like turning “ into ".
-    MS_CHARS_TO_ASCII = {
-        b'\x80' : 'EUR',
-        b'\x81' : ' ',
-        b'\x82' : ',',
-        b'\x83' : 'f',
-        b'\x84' : ',,',
-        b'\x85' : '...',
-        b'\x86' : '+',
-        b'\x87' : '++',
-        b'\x88' : '^',
-        b'\x89' : '%',
-        b'\x8a' : 'S',
-        b'\x8b' : '<',
-        b'\x8c' : 'OE',
-        b'\x8d' : '?',
-        b'\x8e' : 'Z',
-        b'\x8f' : '?',
-        b'\x90' : '?',
-        b'\x91' : "'",
-        b'\x92' : "'",
-        b'\x93' : '"',
-        b'\x94' : '"',
-        b'\x95' : '*',
-        b'\x96' : '-',
-        b'\x97' : '--',
-        b'\x98' : '~',
-        b'\x99' : '(TM)',
-        b'\x9a' : 's',
-        b'\x9b' : '>',
-        b'\x9c' : 'oe',
-        b'\x9d' : '?',
-        b'\x9e' : 'z',
-        b'\x9f' : 'Y',
-        b'\xa0' : ' ',
-        b'\xa1' : '!',
-        b'\xa2' : 'c',
-        b'\xa3' : 'GBP',
-        b'\xa4' : '$', #This approximation is especially parochial--this is the
-                       #generic currency symbol.
-        b'\xa5' : 'YEN',
-        b'\xa6' : '|',
-        b'\xa7' : 'S',
-        b'\xa8' : '..',
-        b'\xa9' : '',
-        b'\xaa' : '(th)',
-        b'\xab' : '<<',
-        b'\xac' : '!',
-        b'\xad' : ' ',
-        b'\xae' : '(R)',
-        b'\xaf' : '-',
-        b'\xb0' : 'o',
-        b'\xb1' : '+-',
-        b'\xb2' : '2',
-        b'\xb3' : '3',
-        b'\xb4' : ("'", 'acute'),
-        b'\xb5' : 'u',
-        b'\xb6' : 'P',
-        b'\xb7' : '*',
-        b'\xb8' : ',',
-        b'\xb9' : '1',
-        b'\xba' : '(th)',
-        b'\xbb' : '>>',
-        b'\xbc' : '1/4',
-        b'\xbd' : '1/2',
-        b'\xbe' : '3/4',
-        b'\xbf' : '?',
-        b'\xc0' : 'A',
-        b'\xc1' : 'A',
-        b'\xc2' : 'A',
-        b'\xc3' : 'A',
-        b'\xc4' : 'A',
-        b'\xc5' : 'A',
-        b'\xc6' : 'AE',
-        b'\xc7' : 'C',
-        b'\xc8' : 'E',
-        b'\xc9' : 'E',
-        b'\xca' : 'E',
-        b'\xcb' : 'E',
-        b'\xcc' : 'I',
-        b'\xcd' : 'I',
-        b'\xce' : 'I',
-        b'\xcf' : 'I',
-        b'\xd0' : 'D',
-        b'\xd1' : 'N',
-        b'\xd2' : 'O',
-        b'\xd3' : 'O',
-        b'\xd4' : 'O',
-        b'\xd5' : 'O',
-        b'\xd6' : 'O',
-        b'\xd7' : '*',
-        b'\xd8' : 'O',
-        b'\xd9' : 'U',
-        b'\xda' : 'U',
-        b'\xdb' : 'U',
-        b'\xdc' : 'U',
-        b'\xdd' : 'Y',
-        b'\xde' : 'b',
-        b'\xdf' : 'B',
-        b'\xe0' : 'a',
-        b'\xe1' : 'a',
-        b'\xe2' : 'a',
-        b'\xe3' : 'a',
-        b'\xe4' : 'a',
-        b'\xe5' : 'a',
-        b'\xe6' : 'ae',
-        b'\xe7' : 'c',
-        b'\xe8' : 'e',
-        b'\xe9' : 'e',
-        b'\xea' : 'e',
-        b'\xeb' : 'e',
-        b'\xec' : 'i',
-        b'\xed' : 'i',
-        b'\xee' : 'i',
-        b'\xef' : 'i',
-        b'\xf0' : 'o',
-        b'\xf1' : 'n',
-        b'\xf2' : 'o',
-        b'\xf3' : 'o',
-        b'\xf4' : 'o',
-        b'\xf5' : 'o',
-        b'\xf6' : 'o',
-        b'\xf7' : '/',
-        b'\xf8' : 'o',
-        b'\xf9' : 'u',
-        b'\xfa' : 'u',
-        b'\xfb' : 'u',
-        b'\xfc' : 'u',
-        b'\xfd' : 'y',
-        b'\xfe' : 'b',
-        b'\xff' : 'y',
-        }
-
-    # A map used when removing rogue Windows-1252/ISO-8859-1
-    # characters in otherwise UTF-8 documents.
-    #
-    # Note that \x81, \x8d, \x8f, \x90, and \x9d are undefined in
-    # Windows-1252.
-    WINDOWS_1252_TO_UTF8 = {
-        0x80 : b'\xe2\x82\xac', # €
-        0x82 : b'\xe2\x80\x9a', # ‚
-        0x83 : b'\xc6\x92',     # ƒ
-        0x84 : b'\xe2\x80\x9e', # „
-        0x85 : b'\xe2\x80\xa6', # …
-        0x86 : b'\xe2\x80\xa0', # †
-        0x87 : b'\xe2\x80\xa1', # ‡
-        0x88 : b'\xcb\x86',     # ˆ
-        0x89 : b'\xe2\x80\xb0', # ‰
-        0x8a : b'\xc5\xa0',     # Š
-        0x8b : b'\xe2\x80\xb9', # ‹
-        0x8c : b'\xc5\x92',     # Œ
-        0x8e : b'\xc5\xbd',     # Ž
-        0x91 : b'\xe2\x80\x98', # ‘
-        0x92 : b'\xe2\x80\x99', # ’
-        0x93 : b'\xe2\x80\x9c', # “
-        0x94 : b'\xe2\x80\x9d', # ”
-        0x95 : b'\xe2\x80\xa2', # •
-        0x96 : b'\xe2\x80\x93', # –
-        0x97 : b'\xe2\x80\x94', # —
-        0x98 : b'\xcb\x9c',     # ˜
-        0x99 : b'\xe2\x84\xa2', # ™
-        0x9a : b'\xc5\xa1',     # š
-        0x9b : b'\xe2\x80\xba', # ›
-        0x9c : b'\xc5\x93',     # œ
-        0x9e : b'\xc5\xbe',     # ž
-        0x9f : b'\xc5\xb8',     # Ÿ
-        0xa0 : b'\xc2\xa0',     #  
-        0xa1 : b'\xc2\xa1',     # ¡
-        0xa2 : b'\xc2\xa2',     # ¢
-        0xa3 : b'\xc2\xa3',     # £
-        0xa4 : b'\xc2\xa4',     # ¤
-        0xa5 : b'\xc2\xa5',     # ¥
-        0xa6 : b'\xc2\xa6',     # ¦
-        0xa7 : b'\xc2\xa7',     # §
-        0xa8 : b'\xc2\xa8',     # ¨
-        0xa9 : b'\xc2\xa9',     # ©
-        0xaa : b'\xc2\xaa',     # ª
-        0xab : b'\xc2\xab',     # «
-        0xac : b'\xc2\xac',     # ¬
-        0xad : b'\xc2\xad',     # ­
-        0xae : b'\xc2\xae',     # ®
-        0xaf : b'\xc2\xaf',     # ¯
-        0xb0 : b'\xc2\xb0',     # °
-        0xb1 : b'\xc2\xb1',     # ±
-        0xb2 : b'\xc2\xb2',     # ²
-        0xb3 : b'\xc2\xb3',     # ³
-        0xb4 : b'\xc2\xb4',     # ´
-        0xb5 : b'\xc2\xb5',     # µ
-        0xb6 : b'\xc2\xb6',     # ¶
-        0xb7 : b'\xc2\xb7',     # ·
-        0xb8 : b'\xc2\xb8',     # ¸
-        0xb9 : b'\xc2\xb9',     # ¹
-        0xba : b'\xc2\xba',     # º
-        0xbb : b'\xc2\xbb',     # »
-        0xbc : b'\xc2\xbc',     # ¼
-        0xbd : b'\xc2\xbd',     # ½
-        0xbe : b'\xc2\xbe',     # ¾
-        0xbf : b'\xc2\xbf',     # ¿
-        0xc0 : b'\xc3\x80',     # À
-        0xc1 : b'\xc3\x81',     # Á
-        0xc2 : b'\xc3\x82',     # Â
-        0xc3 : b'\xc3\x83',     # Ã
-        0xc4 : b'\xc3\x84',     # Ä
-        0xc5 : b'\xc3\x85',     # Å
-        0xc6 : b'\xc3\x86',     # Æ
-        0xc7 : b'\xc3\x87',     # Ç
-        0xc8 : b'\xc3\x88',     # È
-        0xc9 : b'\xc3\x89',     # É
-        0xca : b'\xc3\x8a',     # Ê
-        0xcb : b'\xc3\x8b',     # Ë
-        0xcc : b'\xc3\x8c',     # Ì
-        0xcd : b'\xc3\x8d',     # Í
-        0xce : b'\xc3\x8e',     # Î
-        0xcf : b'\xc3\x8f',     # Ï
-        0xd0 : b'\xc3\x90',     # Ð
-        0xd1 : b'\xc3\x91',     # Ñ
-        0xd2 : b'\xc3\x92',     # Ò
-        0xd3 : b'\xc3\x93',     # Ó
-        0xd4 : b'\xc3\x94',     # Ô
-        0xd5 : b'\xc3\x95',     # Õ
-        0xd6 : b'\xc3\x96',     # Ö
-        0xd7 : b'\xc3\x97',     # ×
-        0xd8 : b'\xc3\x98',     # Ø
-        0xd9 : b'\xc3\x99',     # Ù
-        0xda : b'\xc3\x9a',     # Ú
-        0xdb : b'\xc3\x9b',     # Û
-        0xdc : b'\xc3\x9c',     # Ü
-        0xdd : b'\xc3\x9d',     # Ý
-        0xde : b'\xc3\x9e',     # Þ
-        0xdf : b'\xc3\x9f',     # ß
-        0xe0 : b'\xc3\xa0',     # à
-        0xe1 : b'\xa1',     # á
-        0xe2 : b'\xc3\xa2',     # â
-        0xe3 : b'\xc3\xa3',     # ã
-        0xe4 : b'\xc3\xa4',     # ä
-        0xe5 : b'\xc3\xa5',     # å
-        0xe6 : b'\xc3\xa6',     # æ
-        0xe7 : b'\xc3\xa7',     # ç
-        0xe8 : b'\xc3\xa8',     # è
-        0xe9 : b'\xc3\xa9',     # é
-        0xea : b'\xc3\xaa',     # ê
-        0xeb : b'\xc3\xab',     # ë
-        0xec : b'\xc3\xac',     # ì
-        0xed : b'\xc3\xad',     # í
-        0xee : b'\xc3\xae',     # î
-        0xef : b'\xc3\xaf',     # ï
-        0xf0 : b'\xc3\xb0',     # ð
-        0xf1 : b'\xc3\xb1',     # ñ
-        0xf2 : b'\xc3\xb2',     # ò
-        0xf3 : b'\xc3\xb3',     # ó
-        0xf4 : b'\xc3\xb4',     # ô
-        0xf5 : b'\xc3\xb5',     # õ
-        0xf6 : b'\xc3\xb6',     # ö
-        0xf7 : b'\xc3\xb7',     # ÷
-        0xf8 : b'\xc3\xb8',     # ø
-        0xf9 : b'\xc3\xb9',     # ù
-        0xfa : b'\xc3\xba',     # ú
-        0xfb : b'\xc3\xbb',     # û
-        0xfc : b'\xc3\xbc',     # ü
-        0xfd : b'\xc3\xbd',     # ý
-        0xfe : b'\xc3\xbe',     # þ
-        }
-
-    MULTIBYTE_MARKERS_AND_SIZES = [
-        (0xc2, 0xdf, 2), # 2-byte characters start with a byte C2-DF
-        (0xe0, 0xef, 3), # 3-byte characters start with E0-EF
-        (0xf0, 0xf4, 4), # 4-byte characters start with F0-F4
-        ]
-
-    FIRST_MULTIBYTE_MARKER = MULTIBYTE_MARKERS_AND_SIZES[0][0]
-    LAST_MULTIBYTE_MARKER = MULTIBYTE_MARKERS_AND_SIZES[-1][1]
-
-    @classmethod
-    def detwingle(cls, in_bytes, main_encoding="utf8",
-                  embedded_encoding="windows-1252"):
-        """Fix characters from one encoding embedded in some other encoding.
-
-        Currently the only situation supported is Windows-1252 (or its
-        subset ISO-8859-1), embedded in UTF-8.
-
-        The input must be a bytestring. If you've already converted
-        the document to Unicode, you're too late.
-
-        The output is a bytestring in which `embedded_encoding`
-        characters have been converted to their `main_encoding`
-        equivalents.
-        """
-        if embedded_encoding.replace('_', '-').lower() not in (
-            'windows-1252', 'windows_1252'):
-            raise NotImplementedError(
-                "Windows-1252 and ISO-8859-1 are the only currently supported "
-                "embedded encodings.")
-
-        if main_encoding.lower() not in ('utf8', 'utf-8'):
-            raise NotImplementedError(
-                "UTF-8 is the only currently supported main encoding.")
-
-        byte_chunks = []
-
-        chunk_start = 0
-        pos = 0
-        while pos < len(in_bytes):
-            byte = in_bytes[pos]
-            if not isinstance(byte, int):
-                # Python 2.x
-                byte = ord(byte)
-            if (byte >= cls.FIRST_MULTIBYTE_MARKER
-                and byte <= cls.LAST_MULTIBYTE_MARKER):
-                # This is the start of a UTF-8 multibyte character. Skip
-                # to the end.
-                for start, end, size in cls.MULTIBYTE_MARKERS_AND_SIZES:
-                    if byte >= start and byte <= end:
-                        pos += size
-                        break
-            elif byte >= 0x80 and byte in cls.WINDOWS_1252_TO_UTF8:
-                # We found a Windows-1252 character!
-                # Save the string up to this point as a chunk.
-                byte_chunks.append(in_bytes[chunk_start:pos])
-
-                # Now translate the Windows-1252 character into UTF-8
-                # and add it as another, one-byte chunk.
-                byte_chunks.append(cls.WINDOWS_1252_TO_UTF8[byte])
-                pos += 1
-                chunk_start = pos
-            else:
-                # Go on to the next character.
-                pos += 1
-        if chunk_start == 0:
-            # The string is unchanged.
-            return in_bytes
-        else:
-            # Store the final chunk.
-            byte_chunks.append(in_bytes[chunk_start:])
-        return b''.join(byte_chunks)
-

bs4/testing.py

@@ -1,537 +0,0 @@
-"""Helper classes for tests."""
-
-import copy
-import functools
-import unittest
-from unittest import TestCase
-from bs4 import BeautifulSoup
-from bs4.element import (
-    CharsetMetaAttributeValue,
-    Comment,
-    ContentMetaAttributeValue,
-    Doctype,
-    SoupStrainer,
-)
-
-from bs4.builder import HTMLParserTreeBuilder
-default_builder = HTMLParserTreeBuilder
-
-
-class SoupTest(unittest.TestCase):
-
-    @property
-    def default_builder(self):
-        return default_builder()
-
-    def soup(self, markup, **kwargs):
-        """Build a Beautiful Soup object from markup."""
-        builder = kwargs.pop('builder', self.default_builder)
-        return BeautifulSoup(markup, builder=builder, **kwargs)
-
-    def document_for(self, markup):
-        """Turn an HTML fragment into a document.
-
-        The details depend on the builder.
-        """
-        return self.default_builder.test_fragment_to_document(markup)
-
-    def assertSoupEquals(self, to_parse, compare_parsed_to=None):
-        builder = self.default_builder
-        obj = BeautifulSoup(to_parse, builder=builder)
-        if compare_parsed_to is None:
-            compare_parsed_to = to_parse
-
-        self.assertEqual(obj.decode(), self.document_for(compare_parsed_to))
-
-
-class HTMLTreeBuilderSmokeTest(object):
-
-    """A basic test of a treebuilder's competence.
-
-    Any HTML treebuilder, present or future, should be able to pass
-    these tests. With invalid markup, there's room for interpretation,
-    and different parsers can handle it differently. But with the
-    markup in these tests, there's not much room for interpretation.
-    """
-
-    def assertDoctypeHandled(self, doctype_fragment):
-        """Assert that a given doctype string is handled correctly."""
-        doctype_str, soup = self._document_with_doctype(doctype_fragment)
-
-        # Make sure a Doctype object was created.
-        doctype = soup.contents[0]
-        self.assertEqual(doctype.__class__, Doctype)
-        self.assertEqual(doctype, doctype_fragment)
-        self.assertEqual(str(soup)[:len(doctype_str)], doctype_str)
-
-        # Make sure that the doctype was correctly associated with the
-        # parse tree and that the rest of the document parsed.
-        self.assertEqual(soup.p.contents[0], 'foo')
-
-    def _document_with_doctype(self, doctype_fragment):
-        """Generate and parse a document with the given doctype."""
-        doctype = '<!DOCTYPE %s>' % doctype_fragment
-        markup = doctype + '\n<p>foo</p>'
-        soup = self.soup(markup)
-        return doctype, soup
-
-    def test_normal_doctypes(self):
-        """Make sure normal, everyday HTML doctypes are handled correctly."""
-        self.assertDoctypeHandled("html")
-        self.assertDoctypeHandled(
-            'html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"')
-
-    def test_public_doctype_with_url(self):
-        doctype = 'html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"'
-        self.assertDoctypeHandled(doctype)
-
-    def test_system_doctype(self):
-        self.assertDoctypeHandled('foo SYSTEM "http://www.example.com/"')
-
-    def test_namespaced_system_doctype(self):
-        # We can handle a namespaced doctype with a system ID.
-        self.assertDoctypeHandled('xsl:stylesheet SYSTEM "htmlent.dtd"')
-
-    def test_namespaced_public_doctype(self):
-        # Test a namespaced doctype with a public id.
-        self.assertDoctypeHandled('xsl:stylesheet PUBLIC "htmlent.dtd"')
-
-    def test_real_xhtml_document(self):
-        """A real XHTML document should come out more or less the same as it went in."""
-        markup = b"""<?xml version="1.0" encoding="utf-8"?>
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
-<html xmlns="http://www.w3.org/1999/xhtml">
-<head><title>Hello.</title></head>
-<body>Goodbye.</body>
-</html>"""
-        soup = self.soup(markup)
-        self.assertEqual(
-            soup.encode("utf-8").replace(b"\n", b""),
-            markup.replace(b"\n", b""))
-
-    def test_deepcopy(self):
-        """Make sure you can copy the tree builder.
-
-        This is important because the builder is part of a
-        BeautifulSoup object, and we want to be able to copy that.
-        """
-        copy.deepcopy(self.default_builder)
-
-    def test_p_tag_is_never_empty_element(self):
-        """A <p> tag is never designated as an empty-element tag.
-
-        Even if the markup shows it as an empty-element tag, it
-        shouldn't be presented that way.
-        """
-        soup = self.soup("<p/>")
-        self.assertFalse(soup.p.is_empty_element)
-        self.assertEqual(str(soup.p), "<p></p>")
-
-    def test_unclosed_tags_get_closed(self):
-        """A tag that's not closed by the end of the document should be closed.
-
-        This applies to all tags except empty-element tags.
-        """
-        self.assertSoupEquals("<p>", "<p></p>")
-        self.assertSoupEquals("<b>", "<b></b>")
-
-        self.assertSoupEquals("<br>", "<br/>")
-
-    def test_br_is_always_empty_element_tag(self):
-        """A <br> tag is designated as an empty-element tag.
-
-        Some parsers treat <br></br> as one <br/> tag, some parsers as
-        two tags, but it should always be an empty-element tag.
-        """
-        soup = self.soup("<br></br>")
-        self.assertTrue(soup.br.is_empty_element)
-        self.assertEqual(str(soup.br), "<br/>")
-
-    def test_nested_formatting_elements(self):
-        self.assertSoupEquals("<em><em></em></em>")
-
-    def test_comment(self):
-        # Comments are represented as Comment objects.
-        markup = "<p>foo<!--foobar-->baz</p>"
-        self.assertSoupEquals(markup)
-
-        soup = self.soup(markup)
-        comment = soup.find(text="foobar")
-        self.assertEqual(comment.__class__, Comment)
-
-    def test_preserved_whitespace_in_pre_and_textarea(self):
-        """Whitespace must be preserved in <pre> and <textarea> tags."""
-        self.assertSoupEquals("<pre>   </pre>")
-        self.assertSoupEquals("<textarea> woo  </textarea>")
-
-    def test_nested_inline_elements(self):
-        """Inline elements can be nested indefinitely."""
-        b_tag = "<b>Inside a B tag</b>"
-        self.assertSoupEquals(b_tag)
-
-        nested_b_tag = "<p>A <i>nested <b>tag</b></i></p>"
-        self.assertSoupEquals(nested_b_tag)
-
-        double_nested_b_tag = "<p>A <a>doubly <i>nested <b>tag</b></i></a></p>"
-        self.assertSoupEquals(nested_b_tag)
-
-    def test_nested_block_level_elements(self):
-        """Block elements can be nested."""
-        soup = self.soup('<blockquote><p><b>Foo</b></p></blockquote>')
-        blockquote = soup.blockquote
-        self.assertEqual(blockquote.p.b.string, 'Foo')
-        self.assertEqual(blockquote.b.string, 'Foo')
-
-    def test_correctly_nested_tables(self):
-        """One table can go inside another one."""
-        markup = ('<table id="1">'
-                  '<tr>'
-                  "<td>Here's another table:"
-                  '<table id="2">'
-                  '<tr><td>foo</td></tr>'
-                  '</table></td>')
-
-        self.assertSoupEquals(
-            markup,
-            '<table id="1"><tr><td>Here\'s another table:'
-            '<table id="2"><tr><td>foo</td></tr></table>'
-            '</td></tr></table>')
-
-        self.assertSoupEquals(
-            "<table><thead><tr><td>Foo</td></tr></thead>"
-            "<tbody><tr><td>Bar</td></tr></tbody>"
-            "<tfoot><tr><td>Baz</td></tr></tfoot></table>")
-
-    def test_deeply_nested_multivalued_attribute(self):
-        # html5lib can set the attributes of the same tag many times
-        # as it rearranges the tree. This has caused problems with
-        # multivalued attributes.
-        markup = '<table><div><div class="css"></div></div></table>'
-        soup = self.soup(markup)
-        self.assertEqual(["css"], soup.div.div['class'])
-
-    def test_angle_brackets_in_attribute_values_are_escaped(self):
-        self.assertSoupEquals('<a b="<a>"></a>', '<a b="&lt;a&gt;"></a>')
-
-    def test_entities_in_attributes_converted_to_unicode(self):
-        expect = u'<p id="pi\N{LATIN SMALL LETTER N WITH TILDE}ata"></p>'
-        self.assertSoupEquals('<p id="pi&#241;ata"></p>', expect)
-        self.assertSoupEquals('<p id="pi&#xf1;ata"></p>', expect)
-        self.assertSoupEquals('<p id="pi&ntilde;ata"></p>', expect)
-
-    def test_entities_in_text_converted_to_unicode(self):
-        expect = u'<p>pi\N{LATIN SMALL LETTER N WITH TILDE}ata</p>'
-        self.assertSoupEquals("<p>pi&#241;ata</p>", expect)
-        self.assertSoupEquals("<p>pi&#xf1;ata</p>", expect)
-        self.assertSoupEquals("<p>pi&ntilde;ata</p>", expect)
-
-    def test_quot_entity_converted_to_quotation_mark(self):
-        self.assertSoupEquals("<p>I said &quot;good day!&quot;</p>",
-                              '<p>I said "good day!"</p>')
-
-    def test_out_of_range_entity(self):
-        expect = u"\N{REPLACEMENT CHARACTER}"
-        self.assertSoupEquals("&#10000000000000;", expect)
-        self.assertSoupEquals("&#x10000000000000;", expect)
-        self.assertSoupEquals("&#1000000000;", expect)
-
-    def test_basic_namespaces(self):
-        """Parsers don't need to *understand* namespaces, but at the
-        very least they should not choke on namespaces or lose
-        data."""
-
-        markup = b'<html xmlns="http://www.w3.org/1999/xhtml" xmlns:mathml="http://www.w3.org/1998/Math/MathML" xmlns:svg="http://www.w3.org/2000/svg"><head></head><body><mathml:msqrt>4</mathml:msqrt><b svg:fill="red"></b></body></html>'
-        soup = self.soup(markup)
-        self.assertEqual(markup, soup.encode())
-        html = soup.html
-        self.assertEqual('http://www.w3.org/1999/xhtml', soup.html['xmlns'])
-        self.assertEqual(
-            'http://www.w3.org/1998/Math/MathML', soup.html['xmlns:mathml'])
-        self.assertEqual(
-            'http://www.w3.org/2000/svg', soup.html['xmlns:svg'])
-
-    def test_multivalued_attribute_value_becomes_list(self):
-        markup = b'<a class="foo bar">'
-        soup = self.soup(markup)
-        self.assertEqual(['foo', 'bar'], soup.a['class'])
-
-    #
-    # Generally speaking, tests below this point are more tests of
-    # Beautiful Soup than tests of the tree builders. But parsers are
-    # weird, so we run these tests separately for every tree builder
-    # to detect any differences between them.
-    #
-
-    def test_soupstrainer(self):
-        """Parsers should be able to work with SoupStrainers."""
-        strainer = SoupStrainer("b")
-        soup = self.soup("A <b>bold</b> <meta/> <i>statement</i>",
-                         parse_only=strainer)
-        self.assertEqual(soup.decode(), "<b>bold</b>")
-
-    def test_single_quote_attribute_values_become_double_quotes(self):
-        self.assertSoupEquals("<foo attr='bar'></foo>",
-                              '<foo attr="bar"></foo>')
-
-    def test_attribute_values_with_nested_quotes_are_left_alone(self):
-        text = """<foo attr='bar "brawls" happen'>a</foo>"""
-        self.assertSoupEquals(text)
-
-    def test_attribute_values_with_double_nested_quotes_get_quoted(self):
-        text = """<foo attr='bar "brawls" happen'>a</foo>"""
-        soup = self.soup(text)
-        soup.foo['attr'] = 'Brawls happen at "Bob\'s Bar"'
-        self.assertSoupEquals(
-            soup.foo.decode(),
-            """<foo attr="Brawls happen at &quot;Bob\'s Bar&quot;">a</foo>""")
-
-    def test_ampersand_in_attribute_value_gets_escaped(self):
-        self.assertSoupEquals('<this is="really messed up & stuff"></this>',
-                              '<this is="really messed up &amp; stuff"></this>')
-
-        self.assertSoupEquals(
-            '<a href="http://example.org?a=1&b=2;3">foo</a>',
-            '<a href="http://example.org?a=1&amp;b=2;3">foo</a>')
-
-    def test_escaped_ampersand_in_attribute_value_is_left_alone(self):
-        self.assertSoupEquals('<a href="http://example.org?a=1&amp;b=2;3"></a>')
-
-    def test_entities_in_strings_converted_during_parsing(self):
-        # Both XML and HTML entities are converted to Unicode characters
-        # during parsing.
-        text = "<p>&lt;&lt;sacr&eacute;&#32;bleu!&gt;&gt;</p>"
-        expected = u"<p>&lt;&lt;sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!&gt;&gt;</p>"
-        self.assertSoupEquals(text, expected)
-
-    def test_smart_quotes_converted_on_the_way_in(self):
-        # Microsoft smart quotes are converted to Unicode characters during
-        # parsing.
-        quote = b"<p>\x91Foo\x92</p>"
-        soup = self.soup(quote)
-        self.assertEqual(
-            soup.p.string,
-            u"\N{LEFT SINGLE QUOTATION MARK}Foo\N{RIGHT SINGLE QUOTATION MARK}")
-
-    def test_non_breaking_spaces_converted_on_the_way_in(self):
-        soup = self.soup("<a>&nbsp;&nbsp;</a>")
-        self.assertEqual(soup.a.string, u"\N{NO-BREAK SPACE}" * 2)
-
-    def test_entities_converted_on_the_way_out(self):
-        text = "<p>&lt;&lt;sacr&eacute;&#32;bleu!&gt;&gt;</p>"
-        expected = u"<p>&lt;&lt;sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!&gt;&gt;</p>".encode("utf-8")
-        soup = self.soup(text)
-        self.assertEqual(soup.p.encode("utf-8"), expected)
-
-    def test_real_iso_latin_document(self):
-        # Smoke test of interrelated functionality, using an
-        # easy-to-understand document.
-
-        # Here it is in Unicode. Note that it claims to be in ISO-Latin-1.
-        unicode_html = u'<html><head><meta content="text/html; charset=ISO-Latin-1" http-equiv="Content-type"/></head><body><p>Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!</p></body></html>'
-
-        # That's because we're going to encode it into ISO-Latin-1, and use
-        # that to test.
-        iso_latin_html = unicode_html.encode("iso-8859-1")
-
-        # Parse the ISO-Latin-1 HTML.
-        soup = self.soup(iso_latin_html)
-        # Encode it to UTF-8.
-        result = soup.encode("utf-8")
-
-        # What do we expect the result to look like? Well, it would
-        # look like unicode_html, except that the META tag would say
-        # UTF-8 instead of ISO-Latin-1.
-        expected = unicode_html.replace("ISO-Latin-1", "utf-8")
-
-        # And, of course, it would be in UTF-8, not Unicode.
-        expected = expected.encode("utf-8")
-
-        # Ta-da!
-        self.assertEqual(result, expected)
-
-    def test_real_shift_jis_document(self):
-        # Smoke test to make sure the parser can handle a document in
-        # Shift-JIS encoding, without choking.
-        shift_jis_html = (
-            b'<html><head></head><body><pre>'
-            b'\x82\xb1\x82\xea\x82\xcdShift-JIS\x82\xc5\x83R\x81[\x83f'
-            b'\x83B\x83\x93\x83O\x82\xb3\x82\xea\x82\xbd\x93\xfa\x96{\x8c'
-            b'\xea\x82\xcc\x83t\x83@\x83C\x83\x8b\x82\xc5\x82\xb7\x81B'
-            b'</pre></body></html>')
-        unicode_html = shift_jis_html.decode("shift-jis")
-        soup = self.soup(unicode_html)
-
-        # Make sure the parse tree is correctly encoded to various
-        # encodings.
-        self.assertEqual(soup.encode("utf-8"), unicode_html.encode("utf-8"))
-        self.assertEqual(soup.encode("euc_jp"), unicode_html.encode("euc_jp"))
-
-    def test_real_hebrew_document(self):
-        # A real-world test to make sure we can convert ISO-8859-9 (a
-        # Hebrew encoding) to UTF-8.
-        hebrew_document = b'<html><head><title>Hebrew (ISO 8859-8) in Visual Directionality</title></head><body><h1>Hebrew (ISO 8859-8) in Visual Directionality</h1>\xed\xe5\xec\xf9</body></html>'
-        soup = self.soup(
-            hebrew_document, from_encoding="iso8859-8")
-        self.assertEqual(soup.original_encoding, 'iso8859-8')
-        self.assertEqual(
-            soup.encode('utf-8'),
-            hebrew_document.decode("iso8859-8").encode("utf-8"))
-
-    def test_meta_tag_reflects_current_encoding(self):
-        # Here's the <meta> tag saying that a document is
-        # encoded in Shift-JIS.
-        meta_tag = ('<meta content="text/html; charset=x-sjis" '
-                    'http-equiv="Content-type"/>')
-
-        # Here's a document incorporating that meta tag.
-        shift_jis_html = (
-            '<html><head>\n%s\n'
-            '<meta http-equiv="Content-language" content="ja"/>'
-            '</head><body>Shift-JIS markup goes here.') % meta_tag
-        soup = self.soup(shift_jis_html)
-
-        # Parse the document, and the charset is seemingly unaffected.
-        parsed_meta = soup.find('meta', {'http-equiv': 'Content-type'})
-        content = parsed_meta['content']
-        self.assertEqual('text/html; charset=x-sjis', content)
-
-        # But that value is actually a ContentMetaAttributeValue object.
-        self.assertTrue(isinstance(content, ContentMetaAttributeValue))
-
-        # And it will take on a value that reflects its current
-        # encoding.
-        self.assertEqual('text/html; charset=utf8', content.encode("utf8"))
-
-        # For the rest of the story, see TestSubstitutions in
-        # test_tree.py.
-
-    def test_html5_style_meta_tag_reflects_current_encoding(self):
-        # Here's the <meta> tag saying that a document is
-        # encoded in Shift-JIS.
-        meta_tag = ('<meta id="encoding" charset="x-sjis" />')
-
-        # Here's a document incorporating that meta tag.
-        shift_jis_html = (
-            '<html><head>\n%s\n'
-            '<meta http-equiv="Content-language" content="ja"/>'
-            '</head><body>Shift-JIS markup goes here.') % meta_tag
-        soup = self.soup(shift_jis_html)
-
-        # Parse the document, and the charset is seemingly unaffected.
-        parsed_meta = soup.find('meta', id="encoding")
-        charset = parsed_meta['charset']
-        self.assertEqual('x-sjis', charset)
-
-        # But that value is actually a CharsetMetaAttributeValue object.
-        self.assertTrue(isinstance(charset, CharsetMetaAttributeValue))
-
-        # And it will take on a value that reflects its current
-        # encoding.
-        self.assertEqual('utf8', charset.encode("utf8"))
-
-    def test_tag_with_no_attributes_can_have_attributes_added(self):
-        data = self.soup("<a>text</a>")
-        data.a['foo'] = 'bar'
-        self.assertEqual('<a foo="bar">text</a>', data.a.decode())
-
-class XMLTreeBuilderSmokeTest(object):
-
-    def test_docstring_generated(self):
-        soup = self.soup("<root/>")
-        self.assertEqual(
-            soup.encode(), b'<?xml version="1.0" encoding="utf-8"?>\n<root/>')
-
-    def test_real_xhtml_document(self):
-        """A real XHTML document should come out *exactly* the same as it went in."""
-        markup = b"""<?xml version="1.0" encoding="utf-8"?>