Christophe François authored Feb 11, 2021 and hydrargyrum committed Feb 12, 2021

Json fields can be `null`. In that case, Dict returns None,  which makes
CleanText crash if it's used on the result.

Instead of handling the case each time this happens, we return the
default value (or raise an error if there is none).

Florent Viard authored Jan 13, 2021 and hydrargyrum committed Jan 18, 2021

A metaclass was used with the Dict filter to be able to use it like that:
Dict['hello'] instead of Dict('hello')

It looks like to not be used.

Thus, the code could be simplified, and "special" undocumented usages
avoided. And so this filter will also not have a specificity compared to
other filters.

Rename to the more explicit name cache_is_updatable. Move at class level
so it can be overwritten without having to override __init__.

Since Python 3.4, HTMLParser.unescape is deprecated and replaced with
html.unescape.

Since Python 3.3, some imports have moved from collections to
collections.abc but were kept for compatibility, though compatibility
will be dropped in Python 3.10.
We will keep compatibility ourselves for some time again and thus support
both imports.

François DAILLOUX authored Dec 22, 2020 and hydrargyrum committed Dec 26, 2020

Previously exceptions were all raised on the same line. This breaks external monitoring tools like sentry
that aggregate exceptions by stacktrace.

Because all those exceptions ARE raised at the same line in the code, those tools aggregate them.

The solution here is to raise the exception directly when we decide what type of exception it is.

The code is also easier to read if you raise the exception directly when you find it.

Maxime Gasselin authored Dec 01, 2020 and Damien Mat Jedrzejewski committed Dec 11, 2020

In some case we need to pass attributes between browsers. For example
during a OTP validation.

Quentin Defenouillere authored Nov 12, 2020 and hydrargyrum committed Nov 17, 2020

Example:

```python
amount = Coalesce(
    CleanDecimal.US('//xpath', default=NotAbailable),
    CleanDecimal.US('//xpath2', default=NotAvailable),
)(self.doc)
```

If one of the CleanDecimal returns NotAvailable and the other '0.00', with
the current behavior, Coalesce will crash with "All falsy and no default"
because we use "if value" in the loop of the filter.

With empty(value), Coalesce accepts 0.00 as a valid value for numeric
elements.

Benjamin Bouvier authored Oct 28, 2020 and hydrargyrum committed Nov 17, 2020

Fixes #438



Signed-off-by: Benjamin Bouvier <public@benj.me>

ntome authored Oct 20, 2020 and hydrargyrum committed Oct 28, 2020

HAR is:
- more exploitable, as it's JSON
- more interoperable, as it's pseudo-standard

Drawback is lack of easy access to responses without tools, but
tools/har-to-old.py is here to help.

Augustin Bouverot authored Sep 09, 2020 and hydrargyrum committed Oct 28, 2020

In the TwoFactorBrowser, we couldn't chain authentication method since
the `ValueTransient` were not cleared before leaving the handling method.

Example:
```
self.AUTHENTICATION_METHOD = {
	'sms_otp': self.handle_sms_otp,
	'email': self.handle_email,
}

def init_login(self):
	raise BrowserQuestion(Value('sms_otp'))

def handle_sms_otp(self):
	raise BrowserQuestion(Value('email'))

def handle_email(self):
	self.logged = True
```

This example would do an infinite loop while requesting the email,
because the `sms_otp` value was not cleared, and the authentication
method handler would go for the first `ValueTransient` that is not None,
in that case `sms_otp`

We can now raise a BrowserInteraction exception from an authentication
method that will be handled by another authentication method.

Jerome Berthier authored Oct 14, 2020 and hydrargyrum committed Oct 28, 2020

When using a JsonPage some fields may be None:

{ 'usage': null }

MapIn(Dict('usage', default=''), USAGE_TYPES, NotAvailable)

ntome authored Aug 10, 2020 and hydrargyrum committed Sep 23, 2020

For binary response, HAR allows base64 encoding so data is untouched.
But no such thing exists for binary posted data. Some generators write JSON
with the binary posted data as if it were latin-1 encoded text.
But when reading, what can we do? Think it's binary to encode back to latin-1?
Or think it's text?
HAR gives no answer, so we add a non-standard "x-binary" key to indicate
"text" is actually binary disguised with latin-1.

ntome authored Sep 07, 2020 and hydrargyrum committed Sep 23, 2020

This is a better trick than manually choping the BOM.

ntome authored Sep 01, 2020 and hydrargyrum committed Sep 09, 2020

It's not an optional or costly dependency.

ntome authored Aug 26, 2020 and hydrargyrum committed Sep 09, 2020

Though a BOM makes no sense in UTF-8, some sites insisting on sending one
with JSON data... Drop it then, as standard "json" module fails when it
is present anyway.

Jerome Berthier authored Jul 21, 2020 and hydrargyrum committed Sep 09, 2020

This feature does not modify the API of BrowserSelenium.
To use a remote driver, a config Value 'remote_driver_url'
must be set. If this value is not set the previous behavior
is used, instantiating a local web driver.

The remote driver capabilites are built according to the module
configuration.

ntome authored Aug 27, 2020 and hydrargyrum committed Sep 09, 2020

When switching to regular imports, ItemElementFromAbstractPage will become
obsolete. However, they will not be removed immediately, but they will be
reimplemented to be compatible with the new system. But the reimplementation
works at import-time, not at exec-time, and it's not possible to access params
defined in outer scopes at import-time. So, we make sure the attributes are
explicitly defined, not implicitly.

ntome authored Aug 18, 2020 and hydrargyrum committed Aug 26, 2020

When enabled, the documents is pre-processed so all links of the page are
made absolute using the <base> href (if present) or the page URL.

AbsoluteLink becomes pointless when this option is enabled. It's not enabled
by default as it would break some existing XPaths like:

    starts-with(@HRef, "/foo")

Axel Pelerin authored Aug 12, 2020 and hydrargyrum committed Aug 26, 2020

If we try to create an AbstractPage of another one, the class inherited
will be AbstractPage, in consequence Page. Therefore we need to recursively handle the inheritance.

Lucas Ficheux authored Jul 02, 2020 and hydrargyrum committed Aug 07, 2020

This reverts commit fd0ebef2ca7ea013cf1cc620911b59836d2adee0.