Weboob 1.3

Just before several talks at pycon-fr conference, we are happy to release Weboob version 1.3. A big work has been made to provide a better python 3 support, and we should be able to drop python 2.7 support in few releases (a cleanup has been made for python 2.6, as announced previously).

As usual, CapBank is the most active component of Weboob, with a large rework and improvement of transfer functions. As seen in Changelog, banking modules hava also a high frequency development. From a user point of view, it's important to notice that we disabled auto-update of updates (for a better privacy, see related issue on gitlab for details). All Qt5 applications have also some improvements, and a new QGalleroob application is available.

For developers, several debug tools are very interesting. See for example the dump of current session and highlighting of elements matched by a filter. A workaround for webiste not compatible with OpenSSL (a regular issue) is also available, with the new --nss option on all applications.

Gitlab migration is still not fully done, but we are working on it. New issues are now only accepted on Gitlab, and we will move other tools before next release on it.

Links

Credits

  • Baptiste Delpey
  • Barthélemy Gouby
  • Benjamin Bouvier
  • Benjamin CARTON
  • Benjamin Tampigny
  • Christophe Lampin
  • David Kremer
  • Edouard Lambert
  • Florent Fourcot
  • Florian
  • Florian Duguet
  • François Revol
  • Frédéric Lépy
  • Guillaume Lundy
  • Guillaume Seznec
  • James GALT
  • Jean Walrave
  • Jonathan Schmidt
  • Juliette Fourcot
  • Laurent Bachelier
  • Louis Debeve
  • Matthieu Weber
  • Nicolas Cornu
  • P4ncake
  • Phyks (Lucas Verney)
  • Raymonde Philibert
  • Roger Philibert
  • Romain Bignon
  • Sébastien JEAN
  • Simon Lipp
  • Théo Dorée
  • Vincent Ardisson
  • Vincent Paredes
  • ZeHiro

Changelog

General

  • Big step for python 3 full support (many patches to improve python3 support (core is compatible, a number of modules are compatible))
  • New QGalleroob application for CapImage and CapGallery
  • New CapProfile capability (used in boobank for example)
  • Don't check updates of modules automatically (new --auto-update option)
  • Some code cleanups since python 2.6 is not supported
  • New bibliothequesparis module (CapBook)
  • New billetreduc module (CapCalendarEvent)
  • New blogspot module (CapMessages)
  • New bnppere module (CapBank)
  • New btpbanque module (CapBank)
  • New capeasi module (CapBank)
  • New cices module (CapBank)
  • New cityscoot module (CapDocument)
  • New cmes module (CapBank)
  • New cmmc module (CapBank, CapContact)
  • New ensap module (CapDocument)
  • New erehsbc module (CapBank)
  • New esalia module (CapBank)
  • New foncia module (CapHousing)
  • New lameteoagricole module (CapWeather)
  • New lampiris module (CapDocument)
  • New larousse module (CapTranslate)
  • New linebourse module (CapBank)
  • New myfoncia module (CapDocument)
  • New pixabay module (CapFile, CapImage)
  • New pradoepargne module (CapBank)
  • New ratp module (CapGauge)
  • New tumblr module (CapGallery)
  • New zerobin module (CapPaste)
  • Deleted champslibres module
  • Deleted citelis module
  • Deleted ovs module

General: core

  • Use https for module updates by default
  • Real cleanup of modules folder in installation
  • Force utf-8 for backends config file encoding
  • API change: BrowserPasswordExpired exception now inherit from ActionNeeded (previously from BrowserIncorrectPassword )
  • Print progress on stderr instead of stdout

Documentation

  • Improved module guide

Browser

  • Introduce weboob.browser.nss to replace OpenSSL by libnss
  • Add magic_highlight to open a browser showing an el
  • In debug, write html responses highlighting involved selectors when _highlight_el is set in backend config
  • Enable expiration duration on state
  • URL.open can take headers and methods, like URL.go
  • Support '*' special char in DictElement
  • Print session when debug is enabled
  • Rework of AbstractBrowser build
  • Support headers in stay_or_go method
  • Normalize URL before opening
  • Save content of response after headers, in case there is a crash during call of r.content

Browser: Filters

  • Introduce filters.base, move or alias filters
  • Add AbsoluteLink filter
  • Add FormValue filter
  • Add HasElement filter
  • Add QueryValue filter
  • CleanText(children=False) now reads all text nodes, not just the first one
  • Allow for setting options on CleanHTML filter
  • Fix Type filter when data has already the right data type
  • Add debug() decorator to several filters

Browser: Pages

  • Fix PartialHTMLPage in case of empty content
  • Fix off-by-one error skipping last row on XLSPage

Old Browser

  • Remove support for older mechanize versions

Tools: backend

  • Set fields to NotAvailable instead of NotLoaded when a module does not fill it after fill request.
  • New Module.get_proxy() method

Tools

  • Remove weboob.tools.ordereddict since we now use python>=2.7
  • Remove weboob.tools.compat.check_output since we now use python>=2.7
  • Remove obsolete and unused weboob.tools.property
  • New ValueDate class
  • If a Value is required, it must be set to be valid (in configuration)
  • Value's regexp must now match until end of string

Tools: pdf

  • get_pdf_rows now handles unsorted table lines
  • Support LTCurve objects and output debug images
  • Open method new read as bytes (windows support)

Tools: media_player

  • Use requests instead of urllib2

Capabilities

  • Replace some Field objects by more specialized types (IntField, BoolField, etc)
  • DeltaField now accepts int and long for seconds
  • Copy() of BaseObject now copies fields

CapBank

  • New steps/API to manage transfer (see do_add_recipient, new_recipient and add_recipient)
  • New CapBankTransferAddRecipient to filter out module without relevant methods
  • Use OrderedDict for currencies
  • Add better error codes for TransferError exception
  • Add Pocket class
  • Add Loan class to have more detailed info on loans (not printed today with boobank, only avaiable with weboob API)
  • New code_type field for Investment
  • Add basic tests for banking modules
  • Add sorted_transactions helper
  • New ISIN code checker

CapHousing

  • New 'rooms' and 'bedrooms' and 'utilities' fields for Housing objects

CapVideo

  • Introduce video.ytdl which uses youtube-dl

Applications

  • Support --nss and 'use_nss' in config to replace OpenSSL by NSS
  • Allow user to use '~' when picking log file name
  • Honor quiet option for progress messages

Applications: Qt

  • Add ResultModel for viewing objects hierarchies and FilterTypeMode
  • Some code cleanups/improvements
  • Quit faster by aborting running weboob backend calls
  • Limit results count in ResultsModel
  • Replace urllib.urlopen with requests.get
  • Do not keep references to main windows so that they can be garbage-collected

Applications: boobank

  • Load config at startup
  • New error messages for TransferError childrens
  • Improve OFX formatter

Applications: boobill

  • Reduce number of requests run by download command
  • Add optional PDF conversion on download

Applications: boobcoming

  • Report CANCELLED status

Applications: boobmsg

  • Add a few fillobj calls and export_thread support using numbers

Applications: boobooks

  • Add a new "rented" command

Applications: cookbook

  • Fix unicode comments display in python2

Applications: qcookboob

  • Fix unicode issues
  • Use ID as export filename default

Applications: radioob

  • Fix playlist decoding

Applications: videoob

  • Display xterm-256 color thumbnail
  • Decode utf-8 lines

Formatters: csv

  • Use to_unicode() to be sure it does not fail
  • Fix unicode output in python2

Modules: 750g

  • fix recipe parsing because of website update

Modules: adecco

  • fix parsing job location
  • fix mode site changed
  • Use weboob.tools.compat instead of urllib/urlparse

Modules: afer

  • fix history parsing and transactions date
  • add certificate

Modules: agendaculturel

  • Use weboob.tools.compat instead of urllib/urlparse

Modules: agendadulibre

  • fix french region choice and use https

Modules: allocine

  • fix get_movie_release and pep8
  • bump to browser2
  • Use weboob.tools.compat instead of urllib/urlparse

Modules: allrecipes

  • Use weboob.tools.compat instead of urllib/urlparse

Modules: amazon

  • don't crash anymore on broken links
  • Patch Amazon module
  • Bug fix in payment list (proper)
  • fix order not found crash
  • retry on TooManyRedirects error
  • fixed redirect loop error
  • use xpath "$vars" in a few modules

Modules: amazonstorecard

  • Use weboob.tools.compat instead of urllib/urlparse

Modules: ameli

  • detect more wrong password + improve login state management
  • fixing wrong regexp
  • Use weboob.tools.compat instead of urllib/urlparse
  • fix login validation URL
  • remove some useless ":443" in URLs since weboob normalizes them

Modules: amelipro

  • Use weboob.tools.compat instead of urllib/urlparse
  • remove some useless ":443" in URLs since weboob normalizes them

Modules: americanexpress

  • use url field instead of _link for Accounts
  • parse some other types of accounts
  • sort each transaction page to avoid sorting the whole list
  • set rdate to date
  • handle transactions as deferred
  • coding style
  • fix incorrect date (like 13th month)
  • fixing some accounts
  • fixing xpath on accounts2
  • skip some canceled cards
  • port to python3
  • fixing history and date on deferred
  • detect ActionNeeded
  • handle the new js site along with old site
  • add another fallback for fetching end_of_period

Modules: amundi

  • detect NoAccountsException

Modules: arte

  • do not crash if there is no video in creativ frame + improve test to handle this case
  • fix module / handle date not available in arte cinema json
  • do not handle arte-cinema program submenu
  • fix default 'quality' value
  • fix search and port to Python3
  • base url changed
  • Use weboob.tools.compat instead of urllib/urlparse

Modules: attilasub

  • Use weboob.tools.compat instead of urllib/urlparse

Modules: audioaddict

  • fix Frescaradio + set a default test config in case no one exists
  • bump to browser2 / import unicode_literals
  • remove frescaradio / add classicalradio / There are No public stream available anymore (www.radiotunes.com/founder-letter)
  • fix type in tests

Modules: aum

  • Use weboob.tools.compat instead of urllib/urlparse

Modules: axabanque

  • avoid "is" comparison for strings
  • the history url is not fixed
  • axabanque : adding new browser to handle wealth part
  • fix: handle perp on wealth part
  • fix:: handle bank and wealth part at same time
  • fix: troubles on some accounts / ActionNeeded
  • when accounts have no detailed investments, just return empty
  • use io.BytesIO where deprecated StringIO module was used for bytes
  • investment uses dot decimals and date can be missing
  • do a get request to finish login
  • fixing account type with PEA
  • investments page changed
  • portfolio_share uses comma separator
  • Fix no account and get_form warning
  • fill the diff_percent instead of diff for life insurances
  • avoid relogin after visiting an erroneous wealth page
  • fix url for post
  • Fix recuperation value of invest
  • Fix open for personnal loan and label xpath
  • add isin_code to investments, add history for life insurance accounts
  • handle life insurance history this way only on main website
  • fixing navigation to get rights investments/history
  • deferred card transactions are now flagged as coming
  • improve pagination code
  • sometimes the pagination doesn't work at all
  • iter_coming is not implemented for wealth site
  • basic port to python3
  • pagination of transaction history is broken

Modules: banqueaccord

  • remove dummy test.py files as they are false negatives for weboob_lint
  • use io.BytesIO where deprecated StringIO module was used for bytes
  • html had 2 elements with same id
  • fix exception raised in case of wrong password length
  • fix raise BrowserIncorrectpassword
  • adding missing account type for several banks

Modules: banquepopulaire

  • don't update with an empty token
  • handle new natixis life insurance website
  • fix iter_investement vdate on natixis json
  • set date for investment history and label
  • fix natixis json with missing date
  • some years do not exist as pdf, fallback on json
  • some invest history just have no label
  • don't raise when natixis site is broken
  • retry the backend method in case of unexpected logout
  • minor improvements on retry decorator
  • return investment history in reverse chronological order
  • add another url for natixis error pages
  • fetch quantity of history invest
  • skip transactions without date on natixis
  • handling empty page
  • adding advisor
  • fix 500 errors on natixis website
  • adding entry to match some loans
  • cleaner handling of website crash