Christophe François authored Feb 11, 2021 and hydrargyrum committed Feb 12, 2021

Json fields can be `null`. In that case, Dict returns None,  which makes
CleanText crash if it's used on the result.

Instead of handling the case each time this happens, we return the
default value (or raise an error if there is none).

Sylvie Ye authored Feb 09, 2021 and hydrargyrum committed Feb 12, 2021

and reindent some lines

Gautier P authored Jan 25, 2021 and hydrargyrum committed Jan 27, 2021

Some modules have a strong limitation on phone fetching (leboncoin)

Florent Viard authored Jan 13, 2021 and hydrargyrum committed Jan 18, 2021

A metaclass was used with the Dict filter to be able to use it like that:
Dict['hello'] instead of Dict('hello')

It looks like to not be used.

Thus, the code could be simplified, and "special" undocumented usages
avoided. And so this filter will also not have a specificity compared to
other filters.

Rename to the more explicit name cache_is_updatable. Move at class level
so it can be overwritten without having to override __init__.

Florent Viard authored Dec 09, 2020 and hydrargyrum committed Jan 18, 2021

In the case of an AbstractModule A depending on another AbstractModule
B.
B.CONFIG could be broken if A is loaded before B, for example because an
operation with A is requested first after weboob startup.

The root cause of the issue is that, to load A, the call to its
"__new__" function will rewrite the parents of B on the process but will
not take care to setup it's CONFIG correctly.

Then, if you try to create an instance of B to use it directly, it will
not be considered as an AbstractModule and so it will not be initialized
with AbstractModule.__new__() to setup its CONFIG.

As an example, there is the case of the "caissedepargne" module that
depends on "banquepopulaire", that itself depends on "stet".

This change ensure that, if needed, any intermediary dependency is
properly loaded.

Since Python 3.4, HTMLParser.unescape is deprecated and replaced with
html.unescape.

Since Python 3.3, some imports have moved from collections to
collections.abc but were kept for compatibility, though compatibility
will be dropped in Python 3.10.
We will keep compatibility ourselves for some time again and thus support
both imports.

"No captcha" was the nickname of RecaptchaV2 service, but it's confusing
and less precise. Rename and keep compatibility.

Léo Yvin authored Jan 06, 2021 and hydrargyrum committed Jan 09, 2021

In Value.check_valid method, regexp in validation error message
is not consistent with actual matching test: it uses re.match
and adds ending anchor.
We can "normalize" regexp in objet initialization to be consistent.

Florent Viard authored Jan 04, 2021 and hydrargyrum committed Jan 09, 2021

Allows to match the origin account by iban instead of by id.

Also, show id of the broken parcel, so it's easy to untrack it.

François DAILLOUX authored Dec 22, 2020 and hydrargyrum committed Dec 26, 2020

Previously exceptions were all raised on the same line. This breaks external monitoring tools like sentry
that aggregate exceptions by stacktrace.

Because all those exceptions ARE raised at the same line in the code, those tools aggregate them.

The solution here is to raise the exception directly when we decide what type of exception it is.

The code is also easier to read if you raise the exception directly when you find it.

This raised a RuntimeError.

Maxime Gasselin authored Dec 01, 2020 and Damien Mat Jedrzejewski committed Dec 11, 2020

In some case we need to pass attributes between browsers. For example
during a OTP validation.

Quentin Defenouillere authored Nov 12, 2020 and hydrargyrum committed Nov 17, 2020

Example:

```python
amount = Coalesce(
    CleanDecimal.US('//xpath', default=NotAbailable),
    CleanDecimal.US('//xpath2', default=NotAvailable),
)(self.doc)
```

If one of the CleanDecimal returns NotAvailable and the other '0.00', with
the current behavior, Coalesce will crash with "All falsy and no default"
because we use "if value" in the loop of the filter.

With empty(value), Coalesce accepts 0.00 as a valid value for numeric
elements.

Benjamin Bouvier authored Oct 28, 2020 and hydrargyrum committed Nov 17, 2020

Fixes #438



Signed-off-by: Benjamin Bouvier <public@benj.me>

ntome authored Oct 22, 2020 and hydrargyrum committed Oct 28, 2020

They should replace naked now() calls in modules because those don't fill
the tzinfo attribute and so information is incomplete.

ntome authored Oct 20, 2020 and hydrargyrum committed Oct 28, 2020

HAR is:
- more exploitable, as it's JSON
- more interoperable, as it's pseudo-standard

Drawback is lack of easy access to responses without tools, but
tools/har-to-old.py is here to help.

Augustin Bouverot authored Sep 09, 2020 and hydrargyrum committed Oct 28, 2020

In the TwoFactorBrowser, we couldn't chain authentication method since
the `ValueTransient` were not cleared before leaving the handling method.

Example:
```
self.AUTHENTICATION_METHOD = {
	'sms_otp': self.handle_sms_otp,
	'email': self.handle_email,
}

def init_login(self):
	raise BrowserQuestion(Value('sms_otp'))

def handle_sms_otp(self):
	raise BrowserQuestion(Value('email'))

def handle_email(self):
	self.logged = True
```

This example would do an infinite loop while requesting the email,
because the `sms_otp` value was not cleared, and the authentication
method handler would go for the first `ValueTransient` that is not None,
in that case `sms_otp`

We can now raise a BrowserInteraction exception from an authentication
method that will be handled by another authentication method.

Jerome Berthier authored Oct 14, 2020 and hydrargyrum committed Oct 28, 2020

When using a JsonPage some fields may be None:

{ 'usage': null }

MapIn(Dict('usage', default=''), USAGE_TYPES, NotAvailable)

ntome authored Oct 06, 2020 and hydrargyrum committed Oct 07, 2020

These methods were deprecated in stdlib xml.etree and also in lxml, and
are ultimately removed in python 3.9.

ntome authored Aug 10, 2020 and hydrargyrum committed Sep 23, 2020

For binary response, HAR allows base64 encoding so data is untouched.
But no such thing exists for binary posted data. Some generators write JSON
with the binary posted data as if it were latin-1 encoded text.
But when reading, what can we do? Think it's binary to encode back to latin-1?
Or think it's text?
HAR gives no answer, so we add a non-standard "x-binary" key to indicate
"text" is actually binary disguised with latin-1.

ntome authored Sep 07, 2020 and hydrargyrum committed Sep 23, 2020

This is a better trick than manually choping the BOM.

ntome authored Aug 06, 2020 and hydrargyrum committed Sep 09, 2020

PFMs need to keep track of accounts, which is done using various generic
methods. However, complicated cases may occur on some sites, where no generic
method works, without harming other modules.

A new capability method is introduced to allow module-specific diff behavior.
Essentially, a diff consists in:
- matching accounts, pairs of new/old objects
- new accounts, that were not seen before and are known to be new
- obsolete accounts, that are known to have disappeared
- unknown accounts, which could simply not be matched, but may be mapped by
  another method or manually by a user

Sylvie Ye authored Sep 02, 2020 and hydrargyrum committed Sep 09, 2020

Add theses 2 new attributes in order to be more explicite about transfer behavior:
* can_initiate_transfer_to_untrusted_beneficiary: the module can do transfer to untrusted beneficiary, for example:
        when module can't add new beneficiary without doing a transfer like n26
        or when module can do transfer to a beneficiary not listed in `iter_transfer_recipients` like for PSD2 modules
* can_initiate_transfer_without_emitter: the module can do transfer without giving it the emitter, for example:
        there is only, and will be only, one account like wallet
        or when the module can initiate transfer without emitter and the emitter is chosen afterwards like for PSD2 modules