diff --git a/docs/source/guides/capability.rst b/docs/source/guides/capability.rst index 37d38a5d57f9592897bba776774b41a51e77fb49..da00ce03e7003637ef58f763c5cfb4c4542365cf 100644 --- a/docs/source/guides/capability.rst +++ b/docs/source/guides/capability.rst @@ -5,16 +5,16 @@ A method can raise only its own exceptions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ When you want to return an error, you **must** raise only your own exceptions defined in the capability module. -Never let Python raise his exceptions, for example ``KeyError`` if a parameter given to method isn't found in a local +Never let Python raise his exceptions, for example :py:exc:`KeyError` if a parameter given to method isn't found in a local list. Prefer returning objects ^^^^^^^^^^^^^^^^^^^^^^^^ Python is an object-oriented language, so when your capability supports entities (for example -:class:`weboob.capabilities.video.BaseVideo` with the :class:`weboob.capabilities.video.CapVideo` capability), -you have to create a class derived from :class:`weboob.capabilities.base.BaseObject`, and create an unique method -to get it (for example :func:`get_video() `), instead of several methods like +:class:`~weboob.capabilities.video.BaseVideo` with the :class:`~weboob.capabilities.video.CapVideo` capability), +you have to create a class derived from :py:class:`~weboob.capabilities.base.BaseObject`, and create an unique method +to get it (for example :func:`~weboob.capabilities.video.CapVideo.get_video`), instead of several methods like ``get_video_url()``, ``get_video_preview()``, etc. An object has an unique ID. diff --git a/docs/source/guides/cookbook.rst b/docs/source/guides/cookbook.rst index 13f96176c1cb7504749a89314beae368e981c330..60ee1a24815b52621bf5e1132ae0e5f9697580e8 100644 --- a/docs/source/guides/cookbook.rst +++ b/docs/source/guides/cookbook.rst @@ -27,7 +27,7 @@ Will do:: foo=bar -It also works with :class:`weboob.browser.url.URL` instances:: +It also works with :class:`~weboob.browser.url.URL` instances:: browser.myurl.go(data={'foo': 'bar'}) @@ -98,8 +98,10 @@ Will do:: Have multiple pages on the same URL ----------------------------------- -Define a ``is_here`` attribute in all :class:`weboob.browser.page.Page` class which have the same URL. +Define a ``is_here`` attribute in all :class:`~weboob.browser.pages.Page` class which have the same URL. + The ``is_here`` attr must be a string XPath. + If the XPath is found in the document, the Page will be matched, else other Pages will be checked:: # in browser.py @@ -116,7 +118,7 @@ If the XPath is found in the document, the Page will be matched, else other Page class PageB(HTMLPage): is_here = '//div[text()="Here are the Bs"]' -If an XPath is not enough, `is_here` can be a method returning a bool:: +If an XPath is not enough, ``is_here`` can be a method returning a bool:: class PageA(HTMLPage): def is_here(self): @@ -125,7 +127,8 @@ If an XPath is not enough, `is_here` can be a method returning a bool:: Have a page which is sometimes a ``LoggedPage``, sometimes isn't ---------------------------------------------------------------- -:class:`weboob.browser.pages.LoggedPage` just defines ``logged = True`` while other pages define ``logged = False`` by default. +:class:`~weboob.browser.pages.LoggedPage` just defines ``logged = True`` while other pages define ``logged = False`` by default. + To make this attribute variable, use a ``@property`` method returning a ``bool``:: class HomePage(HTMLPage): @@ -167,7 +170,7 @@ There are multiple ways to skip elements:: Fix invalid HTML that prevents lxml to be parsed ------------------------------------------------ -When the document must be fixed before being parsed, :meth:`weboob.browser.pages.Page.build_doc` can be overridden:: +When the document must be fixed before being parsed, :meth:`~weboob.browser.pages.Page.build_doc` can be overridden:: class MyPage(HTMLPage): def build_doc(self, content): @@ -207,7 +210,7 @@ This example code isn't very semantic and could fail silently if columns are cha obj_id = CleanText('./td[1]') obj_foo = CleanText('./td[2]') -It can be improved by using a :class:`weboob.browser.elements.TableElement` and the column labels:: +It can be improved by using a :class:`~weboob.browser.elements.TableElement` and the column labels:: class MyPage(HTMLPage): @method @@ -231,7 +234,7 @@ Handle multiple tables with similar headers ------------------------------------------- Sometimes, you might encounter a page with multiple tables to parse. The columns are titled the same, but they aren't at the same column index. -So, it's required to restart :class:`weboob.browser.elements.TableElement` column processing for each table. It's possible to encapsulate elements in other elements:: +So, it's required to restart :class:`~weboob.browser.elements.TableElement` column processing for each table. It's possible to encapsulate elements in other elements:: class MultiPage(HTMLPage): @method @@ -335,15 +338,15 @@ Since the pages contain different attributes, info can be merged easily:: return stuff # foo and bar are set -This can also be useful for implementing ``fillobj``. +This can also be useful for implementing :func:`~weboob.tools.backend.Module.fillobj`. -Unfortunately, this doesn't work when multiple objects are parsed (for example, in a ``ListElement``). +Unfortunately, this doesn't work when multiple objects are parsed (for example, in a :class:`~weboob.browser.elements.ListElement`). In this case, manual merging is required, and linking objects from each page. Use ``ItemElement`` with non-scalar attributes ---------------------------------------------- -Some ``BaseObject`` subclasses have fields of other ``BaseObject`` types, for example:: +Some :class:`~weboob.capabilities.base.BaseObject` subclasses have fields of other :class:`~weboob.capabilities.base.BaseObject` types, for example:: class Foo(BaseObject): name = StringField('name') @@ -353,7 +356,7 @@ Some ``BaseObject`` subclasses have fields of other ``BaseObject`` types, for ex foo = Field('foo', Foo) multiple = Field('multiple foo objects', list) -They may still be parsed with ``ItemElement``:: +They may still be parsed with :class:`~weboob.browser.elements.ItemElement`:: class item(ItemElement): klass = Bar @@ -365,7 +368,7 @@ They may still be parsed with ``ItemElement``:: obj_name = CleanText('span') -This also works for ``ListElement``:: +This also works for :class:`~weboob.browser.elements.ListElement`:: class item(ItemElement): klass = Bar diff --git a/docs/source/guides/module.rst b/docs/source/guides/module.rst index 9ab5d86ea4b089e899819d36a067146f40404c5f..897cf89540fc3e980bd148a1a4ed74ba0754ebad 100644 --- a/docs/source/guides/module.rst +++ b/docs/source/guides/module.rst @@ -20,13 +20,13 @@ Select capabilities Each module implements one or many :doc:`capabilities ` to tell what kind of features the website provides. A capability is a class derived from :class:`weboob.capabilities.base.Capability` and with some abstract -methods (which raise ``NotImplementedError``). +methods (which raise :py:exc:`NotImplementedError`). A capability needs to be as generic as possible to allow a maximum number of modules to implement it. Anyway, if you really need to handle website specificities, you can create more specific sub-capabilities. -For example, there is the :class:`CapMessages ` capability, with the associated -:class:`CapMessagesPost ` capability to allow answers to messages. +For example, there is the :class:`~weboob.capabilities.messages.CapMessages` capability, with the associated +:class:`~weboob.capabilities.messages.CapMessagesPost` capability to allow answers to messages. Pick an existing capability --------------------------- @@ -48,7 +48,7 @@ The module tree Create a new directory in ``modules/`` with the name of your module. In this example, we assume that we want to create a module for a bank website which URL is http://www.example.com. So we will call our module **example**, and the selected -capability is :class:`CapBank `. +capability is :class:`~weboob.capabilities.bank.CapBank`. It is recommended to use the helper tool ``tools/boilerplate.py`` to build your module tree. There are several templates available: @@ -63,9 +63,9 @@ For example, use this command:: In a module directory, there are commonly these files: -* **__init__.py** - needed in every python modules, it exports your :class:`Module ` class. -* **module.py** - defines the main class of your module, which derives :class:`Module `. -* **browser.py** - your browser, derived from :class:`Browser `, is called by your module to interact with the supported website. +* **__init__.py** - needed in every python modules, it exports your :class:`~weboob.tools.backend.Module` class. +* **module.py** - defines the main class of your module, which derives :class:`~weboob.tools.backend.Module`. +* **browser.py** - your browser, derived from :class:`~weboob.browser.browsers.Browser`, is called by your module to interact with the supported website. * **pages.py** - all website's pages handled by the browser are defined here * **test.py** - functional tests * **favicon.png** - a 64x64 transparent PNG icon @@ -109,7 +109,7 @@ Weboob applications to use local modules rather than remote ones. Module class ************* -Edit ``module.py``. It contains the main class of the module derived from :class:`Module ` class:: +Edit ``module.py``. It contains the main class of the module derived from :class:`~weboob.tools.backend.Module` class:: from weboob.tools.backend import Module from weboob.capabilities.bank import CapBank @@ -123,17 +123,17 @@ Edit ``module.py``. It contains the main class of the module derived from :class # Version of weboob VERSION = '2.1' -In the code above, you can see that your ``ExampleModule`` inherits :class:`CapBank `, as +In the code above, you can see that your ``ExampleModule`` inherits :class:`~weboob.capabilities.bank.base.CapBank`, as we have selected it for the supported website. Configuration ------------- When a module is instanced as a backend, you probably want to ask parameters to user. It is managed by the ``CONFIG`` class -attribute. It supports key/values with default values and some other parameters. The :class:`Value ` +attribute. It supports key/values with default values and some other parameters. The :class:`~weboob.tools.value.Value` class is used to define a value. -Available parameters of :class:`Value ` are: +Available parameters of :class:`~weboob.tools.value.Value` are: * **label** - human readable description of a value * **required** - if ``True``, the backend can't be loaded if the key isn't found in its configuration @@ -146,7 +146,7 @@ Available parameters of :class:`Value ` are: .. note:: - There is a special class, :class:`ValueBackendPassword `, which is used to manage + There is a special class, :class:`~weboob.tools.value.ValueBackendPassword`, which is used to manage private parameters of the config (like passwords or sensitive information). .. note:: @@ -198,7 +198,7 @@ You need to implement each method of all of the capabilities your module impleme If you ran the ``boilerplate`` script command ``cap``, every methods are already in ``module.py`` and documented. -Read :class:`documentation of the capability ` to know what are types of arguments, +Read :class:`documentation of the capability ` to know what are types of arguments, what are expected returned objects, and what exceptions it may raise. When you are done writing your module, you should remove all the not implemented methods from your module, as the base @@ -208,9 +208,9 @@ capability code will anyway ``raise NotImplementedError()``. Browser ******* -Most of modules use a class derived from :class:`PagesBrowser ` or -:class:`LoginBrowser ` (for authenticated websites) to interact with a website or -:class:`APIBrowser ` to interact with an API. +Most of modules use a class derived from :class:`~weboob.browser.browsers.PagesBrowser` or +:class:`~weboob.browser.browsers.LoginBrowser` (for authenticated websites) to interact with a website or +:class:`~weboob.browser.browsers.APIBrowser` to interact with an API. Edit ``browser.py``:: @@ -225,7 +225,7 @@ Edit ``browser.py``:: There are several possible class attributes: -* **BASEURL** - base url of website used for absolute paths given to :class:`open() ` or :class:`location() ` +* **BASEURL** - base url of website used for absolute paths given to :class:`~weboob.browser.browsers.PagesBrowser.open` or :class:`~weboob.browser.browsers.PagesBrowser.location` * **PROFILE** - defines the behavior of your browser against the website. By default this is Firefox, but you can import other profiles * **TIMEOUT** - defines the timeout for requests (defaults to 10 seconds) * **VERIFY** - SSL verification (if the protocol used is **https**) @@ -235,10 +235,10 @@ Pages For each page you want to handle, you have to create an associated class derived from one of these classes: -* :class:`HTMLPage ` - a HTML page -* :class:`XMLPage ` - a XML document -* :class:`JsonPage ` - a Json object -* :class:`CsvPage ` - a CSV table +* :class:`~weboob.browser.pages.HTMLPage` - a HTML page +* :class:`~weboob.browser.pages.XMLPage` - a XML document +* :class:`~weboob.browser.pages.JsonPage` - a Json object +* :class:`~weboob.browser.pages.CsvPage` - a CSV table In the file ``pages.py``, you can write, for example:: @@ -258,7 +258,7 @@ In the file ``pages.py``, you can write, for example:: ``IndexPage`` is the class we will use to get information from the home page of the website, and ``ListPage`` will handle pages which list accounts. -Then, you have to declare them in your browser, with the :class:`URL ` object:: +Then, you have to declare them in your browser, with the :class:`~weboob.browser.url.URL` object:: from weboob.browser import PagesBrowser, URL from .pages import IndexPage, ListPage @@ -275,7 +275,7 @@ Easy, isn't it? The first parameters are regexps of the urls (if you give only a .. note:: You can handle parameters in the URL using ``(?P)``. You can then use a keyword argument `someName` to - bind a value to this parameter in :func:`stay_or_go() `. + bind a value to this parameter in :func:`~weboob.browser.url.URL.stay_or_go`. Each time you will go on the home page, ``IndexPage`` will be instanced and set as the ``page`` attribute. @@ -295,9 +295,9 @@ For example, we can now implement some methods in ``ExampleBrowser``:: return self.page.iter_accounts() -When calling the :func:`go() ` method, it reads the first regexp url of our :class:`URL ` object, and go on the page. +When calling the :func:`~weboob.browser.url.URL.go` method, it reads the first regexp url of our :class:`~weboob.browser.url.URL` object, and go on the page. -:func:`stay_or_go() ` is used when you want to relocate on the page only if we aren't already on it. +:func:`~weboob.browser.url.URL.stay_or_go` is used when you want to relocate on the page only if we aren't already on it. Once we are on the ``ListPage``, we can call every methods of the ``page`` object. @@ -318,7 +318,7 @@ Now you have a functional browser, you can use it in your class ``ExampleModule` You can now access it with member ``browser``. The class is instanced at the first call to this attribute. -For example, we can now implement :func:`CapBank.iter_accounts `:: +For example, we can now implement :func:`CapBank.iter_accounts `:: def iter_accounts(self): return self.browser.iter_accounts_list() @@ -329,7 +329,7 @@ Login management ---------------- When the website requires to be authenticated, you have to give credentials to the constructor of the browser. You can redefine -the method :func:`create_default_browser `:: +the method :func:`~weboob.tools.backend.Module.create_default_browser`:: from weboob.tools.backend import Module from weboob.capabilities.bank import CapBank @@ -339,8 +339,8 @@ the method :func:`create_default_browser ` and to implement the function -:func:`do_login `:: +On the browser side, you need to inherit from :func:`~weboob.browser.browsers.LoginBrowser` and to implement the function +:func:`~weboob.browser.browsers.LoginBrowser.do_login`:: from weboob.browser import LoginBrowser from weboob.exceptions import BrowserIncorrectPassword @@ -357,7 +357,7 @@ On the browser side, you need to inherit from :func:`LoginBrowser `:: function if you need to customize the default logout process, which simply clears all cookies. +You may provide a custom :func:`~weboob.browser.browsers.LoginBrowser.do_logout` function if you need to customize the default logout process, which simply clears all cookies. Also, your ``LoginPage`` may look like:: @@ -370,7 +370,7 @@ Also, your ``LoginPage`` may look like:: form['password'] = password form.submit() -Then, each method on your browser which needs your user to be authenticated may be decorated by :func:`need_login `:: +Then, each method on your browser which needs your user to be authenticated may be decorated by :func:`~weboob.browser.browsers.need_login`:: from weboob.browser import LoginBrowser, URL from weboob.browser import need_login @@ -383,14 +383,13 @@ Then, each method on your browser which needs your user to be authenticated may self.accounts.stay_or_go() return self.page.get_accounts() -You finally have to set correctly the :func:`logged ` attribute of each page you use. The -:func:`need_login ` decorator checks if the current page is a logged one by reading the attribute -:func:`logged ` of the instance. This attributes defaults to ``False``, which means that :func:`need_login -` will first call :func:`do_login ` before calling the +You finally have to set correctly the :func:`~weboob.browser.pages.Page.logged` attribute of each page you use. The +:func:`~weboob.browser.browsers.need_login` decorator checks if the current page is a logged one by reading the attribute +:func:`~weboob.browser.pages.Page.logged` of the instance. This attributes defaults to ``False``, which means that :func:`~weboob.browser.browsers.need_login` will first call :func:`~weboob.browser.browsers.LoginBrowser.do_login` before calling the decorated method. -You can either define it yourself, as a class boolean attribute or as a property, or inherit your class from :class:`LoggedPage `. -In the latter case, remember that Python inheritance requires the :class:`LoggedPage ` to be placed first such as in:: +You can either define it yourself, as a class boolean attribute or as a property, or inherit your class from :class:`~weboob.browser.pages.LoggedPage`. +In the latter case, remember that Python inheritance requires the :class:`~weboob.browser.pages.LoggedPage` to be placed first such as in:: from weboob.browser.pages import LoggedPage, HTMLPage @@ -407,7 +406,7 @@ Parsing of pages When your browser locates on a page, an instance of the class related to the -:class:`URL ` attribute which matches the url +:class:`~weboob.browser.url.URL` attribute which matches the url is created. You can declare methods on your class to allow your browser to interact with it. @@ -443,20 +442,20 @@ either be a filter, a constant or a function. Some example of filters: -* :class:`Attr `: extract a tag attribute -* :class:`CleanText `: get a cleaned text from an element -* :class:`CleanDecimal `: get a cleaned Decimal value from an element -* :class:`Date `: read common date formats -* :class:`DateTime `: read common datetime formats -* :class:`Env `: typically useful to get a named parameter in the URL (passed as a - keyword argument to :func:`stay_or_go() `) -* :class:`Eval `: evaluate a lambda on the given value -* :class:`Format `: a formatting filter, uses the standard Python format string +* :class:`~weboob.browser.filters.html.Attr`: extract a tag attribute +* :class:`~weboob.browser.filters.standard.CleanText`: get a cleaned text from an element +* :class:`~weboob.browser.filters.standard.CleanDecimal`: get a cleaned Decimal value from an element +* :class:`~weboob.browser.filters.standard.Date`: read common date formats +* :class:`~weboob.browser.filters.standard.DateTime`: read common datetime formats +* :class:`~weboob.browser.filters.standard.Env`: typically useful to get a named parameter in the URL (passed as a + keyword argument to :func:`~weboob.browser.url.URL.stay_or_go`) +* :class:`~weboob.browser.filters.standard.Eval`: evaluate a lambda on the given value +* :class:`~weboob.browser.filters.standard.Format`: a formatting filter, uses the standard Python format string notations. -* :class:`Link `: get the link uri of an element -* :class:`Regexp `: apply a regex -* :class:`Time `: read common time formats -* :class:`Type `: get a cleaned value of any type from an element text +* :class:`~weboob.browser.filters.html.Link`: get the link uri of an element +* :class:`~weboob.browser.filters.standard.Regexp`: apply a regex +* :class:`~weboob.browser.filters.standard.Time`: read common time formats +* :class:`~weboob.browser.filters.standard.Type`: get a cleaned value of any type from an element text The full list of filters can be found in :doc:`weboob.browser.filters `. @@ -471,7 +470,7 @@ This code do several things, in order: #) convert this value to int type -When you want to access some attributes of your :class:`HTMLPage ` object to fill an +When you want to access some attributes of your :class:`~weboob.browser.pages.HTMLPage` object to fill an attribute in a Filter, you should use the function construction for this attribute. For example:: def obj_url(self): @@ -537,19 +536,19 @@ Filling objects .. note:: - Filling objects using ``fillobj`` should be used whenever you need to fill some fields automatically based on data + Filling objects using :func:`~weboob.tools.backend.Module.fillobj` should be used whenever you need to fill some fields automatically based on data fetched from the scraping. If you only want to fill some fields automatically based on some static data, you should just inherit the base object class and set these fields. An object returned by a method of a capability can be not fully completed. -The class :class:`Module ` provides a method named -:func:`fillobj `, which can be called by an application to +The class :class:`~weboob.tools.backend.Module` provides a method named +:func:`~weboob.tools.backend.Module.fillobj`, which can be called by an application to fill some unloaded fields of a specific object, for example with:: backend.fillobj(video, ['url', 'author']) -The ``fillobj`` method will check on the object which fields (in the ones given in the list argument) are not loaded +The :func:`~weboob.tools.backend.Module.fillobj` method will check on the object which fields (in the ones given in the list argument) are not loaded (equal to ``NotLoaded``, which is the default value), to reduce the list to the real uncompleted fields, and call the method associated to the type of the object. @@ -583,9 +582,9 @@ Then, the function might, for each requested fields, fetch the right data and fi return video Here, when the application has got a :class:`Video ` object with -:func:`search_videos `, in most cases, there are only some meta-data, but not the direct link to the video media. +:func:`~weboob.capabilities.video.CapVideo.search_videos`, in most cases, there are only some meta-data, but not the direct link to the video media. -As our method :func:`get_video ` will get all +As our method :func:`~weboob.capabilities.video.CapVideo.get_video` will get all of the missing data, we just call it with the object as parameter to complete it. @@ -596,7 +595,7 @@ The application can provide a storage to let your backend store data. So, you ca STORAGE = {'seen': {}} -To store and read data in your storage space, use the ``storage`` attribute of your :class:`Module ` +To store and read data in your storage space, use the ``storage`` attribute of your :class:`~weboob.tools.backend.Module` object. -It implements the methods of :class:`BackendStorage `. +It implements the methods of :class:`~weboob.tools.backend.BackendStorage`. diff --git a/docs/source/guides/tests.rst b/docs/source/guides/tests.rst index 32a8daf86fdf037b0f9c51d9445f2e5d8be89f56..3cf387bbcadf3a82ba0eb6dcbf3d18cab9b3b1a6 100644 --- a/docs/source/guides/tests.rst +++ b/docs/source/guides/tests.rst @@ -37,7 +37,7 @@ Write a test case Normal test ----------- -Use the class :class:`TestCase ` to derivate it into your new test case. Then, write methods which name starts with ``test_``. +Use the class :class:`~weboob.tools.test.TestCase` to derivate it into your new test case. Then, write methods which name starts with ``test_``. A test fails when an assertion error is raised. Also, when an other kind of exception is raised, this is an error. @@ -46,7 +46,7 @@ You can use ``assert`` to check something, or the base methods ``assertTrue``, ` Backend test ------------ -Create a class derivated from :class:`BackendTest `, and set the ``BACKEND`` class attribute to the name of the backend to test. +Create a class derivated from :class:`~weboob.tools.test.BackendTest`, and set the ``BACKEND`` class attribute to the name of the backend to test. Then, in your test methods, the ``backend`` attribute will contain the loaded backend. When the class is instancied, it loads every configured backends of the right type, and randomly choose one. If no one is found, the tests are skipped.