module.rst 24.2 KB
Newer Older
Romain Bignon's avatar
Romain Bignon committed
1 2 3 4 5 6 7 8 9 10
Write a new module
==================

This guide aims to learn how to write a new module for `Weboob <http://weboob.org>`_.

Before read it, you should :doc:`setup your development environment </guides/setup>`.

What is a module
****************

11
A module is an interface between a website and Weboob. It represents the python code which is stored
Romain Bignon's avatar
Romain Bignon committed
12 13
in repositories.

14
Weboob applications need *backends* to interact with websites. A *backend* is an instance of a *module*, usually
Romain Bignon's avatar
Romain Bignon committed
15 16 17 18 19 20 21
with several parameters like your username, password, or other options. You can create multiple *backends*
for a single *module*.

Select capabilities
*******************

Each module implements one or many :doc:`capabilities </api/capabilities/index>` to tell what kind of features the
22
website provides. A capability is a class derived from :class:`weboob.capabilities.base.Capability` and with some abstract
Romain Bignon's avatar
Romain Bignon committed
23 24
methods (which raise ``NotImplementedError``).

25
A capability needs to be as generic as possible to allow a maximum number of modules to implement it.
Romain Bignon's avatar
Romain Bignon committed
26 27
Anyway, if you really need to handle website specificities, you can create more specific sub-capabilities.

28 29
For example, there is the :class:`CapMessages <weboob.capabilities.messages.CapMessages>` capability, with the associated
:class:`CapMessagesPost <weboob.capabilities.messages.CapMessagesPost>` capability to allow answers to messages.
Romain Bignon's avatar
Romain Bignon committed
30 31 32 33

Pick an existing capability
---------------------------

34
When you want to create a new module, you may have a look at existing capabilities to decide which one can be
Romain Bignon's avatar
Romain Bignon committed
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
implemented. It is quite important, because each already existing capability is supported by at least one application.
So if your new module implements an existing capability, it will be usable from the existing applications right now.

Create a new capability
-----------------------

If the website you want to manage implements some extra-features which are not implemented by any capability,
you can introduce a new capability.

You should read the related guide to know :doc:`how to create a capability </guides/capability>`.

The module tree
***************

Create a new directory in ``modules/`` with the name of your module. In this example, we assume that we want to create a
50 51
module for a bank website which URL is http://www.example.com. So we will call our module **example**, and the selected
capability is :class:`CapBank <weboob.capabilities.bank.CapBank>`.
Romain Bignon's avatar
Romain Bignon committed
52

53 54
It is recommended to use the helper tool ``tools/boilerplate.py`` to build your
module tree. There are several templates available:
Romain Bignon's avatar
Romain Bignon committed
55

56 57 58 59 60 61 62
* **base** - create only base files
* **comic** - create a comic module
* **cap** - create a module for a given capability

For example, use this command::

    $ tools/boilerplate.py cap example CapBank
Romain Bignon's avatar
Romain Bignon committed
63 64 65

In a module directory, there are commonly these files:

66 67
* **__init__.py** - needed in every python modules, it exports your :class:`Module <weboob.tools.backend.Module>` class.
* **module.py** - defines the main class of your module, which derives :class:`Module <weboob.tools.backend.Module>`.
68
* **browser.py** - your browser, derived from :class:`Browser <weboob.browser.browsers.Browser>`, is called by your module to interact with the supported website.
Romain Bignon's avatar
Romain Bignon committed
69 70
* **pages.py** - all website's pages handled by the browser are defined here
* **test.py** - functional tests
71
* **favicon.png** - a 64x64 transparent PNG icon
Romain Bignon's avatar
Romain Bignon committed
72

73 74 75 76 77 78
.. note::

    A module can implement multiple capabilities, even though the ``tools/boilerplate.py`` script can only generate a
    template for a single capability. You can freely add inheritance from other capabilities afterwards in
    ``module.py``.

Romain Bignon's avatar
Romain Bignon committed
79 80 81 82 83
Update modules list
-------------------

As you are in development mode, to see your new module in ``weboob-config``'s list, you have to update ``modules/modules.list`` with this command::

84
    $ weboob update
Romain Bignon's avatar
Romain Bignon committed
85 86 87 88 89 90 91

To be sure your module is correctly added, use this command::

    $ weboob-config info example
    .------------------------------------------------------------------------------.
    | Module example                                                               |
    +-----------------.------------------------------------------------------------'
92
    | Version         | 201405191420
Romain Bignon's avatar
Romain Bignon committed
93
    | Maintainer      | John Smith <john.smith@example.com>
94
    | License         | LGPLv3+
95 96
    | Description     | Example bank website
    | Capabilities    | CapBank, CapCollection
Romain Bignon's avatar
Romain Bignon committed
97 98 99 100
    | Installed       | yes
    | Location        | /home/me/src/weboob/modules/example
    '-----------------'

101 102 103 104 105 106 107
If the last command does not work, check your :doc:`repositories setup
</guides/setup>`. In particular, when you want to edit an already existing
module, you should take great care of setting your development environment
correctly, or your changes to the module will not have any effect. You can also
use ``./tools/local_run.sh`` script as a quick and dirty method of forcing
Weboob applications to use local modules rather than remote ones.

108

109
Module class
110 111
*************

112
Edit ``module.py``. It contains the main class of the module derived from :class:`Module <weboob.tools.backend.Module>` class::
113

114 115 116
    from weboob.tools.backend import Module
    from weboob.capabilities.bank import CapBank

117
    class ExampleModule(Module, CapBank):
118 119 120 121
        NAME = 'example'                         # The name of module
        DESCRIPTION = u'Example bank website'    # Description of your module
        MAINTAINER = u'John Smith'               # Name of maintainer of this module
        EMAIL = 'john.smith@example.com'         # Email address of the maintainer
122
        LICENSE = 'LGPLv3+'                      # License of your module
123
        # Version of weboob
Romain Bignon's avatar
Romain Bignon committed
124
        VERSION = '1.5'
125

126
In the code above, you can see that your ``ExampleModule`` inherits :class:`CapBank <weboob.capabilities.bank.CapBank>`, as
127 128
we have selected it for the supported website.

Romain Bignon's avatar
Romain Bignon committed
129 130 131
Configuration
-------------

132
When a module is instanced as a backend, you probably want to ask parameters to user. It is managed by the ``CONFIG`` class
Romain Bignon's avatar
Romain Bignon committed
133 134 135
attribute. It supports key/values with default values and some other parameters. The :class:`Value <weboob.tools.value.Value>`
class is used to define a value.

136
Available parameters of :class:`Value <weboob.tools.value.Value>` are:
Romain Bignon's avatar
Romain Bignon committed
137 138

* **label** - human readable description of a value
139
* **required** - if ``True``, the backend can't be loaded if the key isn't found in its configuration
Romain Bignon's avatar
Romain Bignon committed
140 141 142
* **default** - an optional default value, used when the key is not in config. If there is no default value and the key
  is not found in configuration, the **required** parameter is implicitly set
* **masked** - if ``True``, the value is masked. It is useful for applications to know if this key is a password
143 144
* **regexp** - if specified, the specified value is checked against this regexp upon loading, and an error is raised if
  it doesn't match
Romain Bignon's avatar
Romain Bignon committed
145 146
* **choices** - if this parameter is set, the value must be in the list

147 148 149 150
.. note::

    There is a special class, :class:`ValueBackendPassword <weboob.tools.value.ValueBackendPassword>`, which is used to manage
    private parameters of the config (like passwords or sensible information).
Romain Bignon's avatar
Romain Bignon committed
151

152 153 154 155 156
.. note::

    Other classes are available to store specific types of configuration options. See :mod:`weboob.tools.value
    <weboob.tools.value>` for a full list of them.

Romain Bignon's avatar
Romain Bignon committed
157 158
For example::

159 160
    from weboob.tools.backend import Module, BackendConfig
    from weboob.capabilities.bank import CapBank
Romain Bignon's avatar
Romain Bignon committed
161 162 163
    from weboob.tools.value import Value, ValueBool, ValueInt, ValueBackendPassword

    # ...
164
    class ExampleModule(Module, CapBank):
Romain Bignon's avatar
Romain Bignon committed
165 166 167 168 169 170 171 172 173 174 175 176 177 178 179
        # ...
        CONFIG = BackendConfig(Value('username',                label='Username', regexp='.+'),
                               ValueBackendPassword('password', label='Password'),
                               ValueBool('get_news',            label='Get newspapers', default=True),
                               Value('choice',                  label='Choices', choices={'value1': 'Label 1',
                                                                                          'value2': 'Label 2'}, default='1'),
                               Value('regexp',                  label='Birthday', regexp='^\d+/\d+/\d+$'),
                               ValueInt('integer',              label='A number', required=True))


Implement capabilities
----------------------

You need to implement each method of all of the capabilities your module implements. For example, in our case::

180 181 182
    from weboob.tools.backend import Module
    from weboob.capabilities.bank import CapBank

Romain Bignon's avatar
Romain Bignon committed
183
    # ...
184
    class ExampleModule(Module, CapBank):
Romain Bignon's avatar
Romain Bignon committed
185 186
        # ...

187
        def iter_accounts(self):
Romain Bignon's avatar
Romain Bignon committed
188 189
            raise NotImplementedError()

190
        def get_account(self, id):
Romain Bignon's avatar
Romain Bignon committed
191 192
            raise NotImplementedError()

193
        def iter_history(self, account):
Romain Bignon's avatar
Romain Bignon committed
194 195
            raise NotImplementedError()

196
        def iter_coming(self, account):
Romain Bignon's avatar
Romain Bignon committed
197 198
            raise NotImplementedError()

199
If you ran the ``boilerplate`` script command ``cap``, every methods are already in ``module.py`` and documented.
200 201

Read :class:`documentation of the capability <weboob.capabilities.bank.CapBank>` to know what are types of arguments,
202 203 204 205
what are expected returned objects, and what exceptions it may raise.

When you are done writing your module, you should remove all the not implemented methods from your module, as the base
capability code will anyway ``raise NotImplementedError()``.
Romain Bignon's avatar
Romain Bignon committed
206 207 208 209 210


Browser
*******

211
Most of modules use a class derived from :class:`PagesBrowser <weboob.browser.browsers.PagesBrowser>` or
212 213
:class:`LoginBrowser <weboob.browser.browsers.LoginBrowser>` (for authenticated websites) to interact with a website or
:class:`APIBrowser <weboob.browser.browsers.APIBrowser>` to interact with an API.
Romain Bignon's avatar
Romain Bignon committed
214

215
Edit ``browser.py``::
Romain Bignon's avatar
Romain Bignon committed
216 217 218

    # -*- coding: utf-8 -*-

219
    from weboob.browser import PagesBrowser
Romain Bignon's avatar
Romain Bignon committed
220 221 222

    __all__ = ['ExampleBrowser']

223 224
    class ExampleBrowser(PagesBrowser):
        BASEURL = 'https://www.example.com'
Romain Bignon's avatar
Romain Bignon committed
225

226
There are several possible class attributes:
Romain Bignon's avatar
Romain Bignon committed
227

228
* **BASEURL** - base url of website used for absolute paths given to :class:`open() <weboob.browser.browsers.PagesBrowser.open>` or :class:`location() <weboob.browser.browsers.PagesBrowser.location>`
229 230 231
* **PROFILE** - defines the behavior of your browser against the website. By default this is Firefox, but you can import other profiles
* **TIMEOUT** - defines the timeout for requests (defaults to 10 seconds)
* **VERIFY** - SSL verification (if the protocol used is **https**)
Romain Bignon's avatar
Romain Bignon committed
232 233 234 235

Pages
-----

236 237
For each page you want to handle, you have to create an associated class derived from one of these classes:

238 239 240 241
* :class:`HTMLPage <weboob.browser.pages.HTMLPage>` - a HTML page
* :class:`XMLPage <weboob.browser.pages.XMLPage>` - a XML document
* :class:`JsonPage <weboob.browser.pages.JsonPage>` - a Json object
* :class:`CsvPage <weboob.browser.pages.CsvPage>` - a CSV table
Romain Bignon's avatar
Romain Bignon committed
242

243
In the file ``pages.py``, you can write, for example::
Romain Bignon's avatar
Romain Bignon committed
244 245 246

    # -*- coding: utf-8 -*-

247
    from weboob.browser.pages import HTMLPage
Romain Bignon's avatar
Romain Bignon committed
248 249 250

    __all__ = ['IndexPage', 'ListPage']

251
    class IndexPage(HTMLPage):
Romain Bignon's avatar
Romain Bignon committed
252 253
        pass

254 255
    class ListPage(HTMLPage):
        def iter_accounts():
Romain Bignon's avatar
Romain Bignon committed
256 257 258
            return iter([])

``IndexPage`` is the class we will use to get information from the home page of the website, and ``ListPage`` will handle pages
259
which list accounts.
Romain Bignon's avatar
Romain Bignon committed
260

261
Then, you have to declare them in your browser, with the :class:`URL <weboob.browser.url.URL>` object::
262

263
    from weboob.browser import PagesBrowser, URL
Romain Bignon's avatar
Romain Bignon committed
264 265 266
    from .pages import IndexPage, ListPage

    # ...
267
    class ExampleBrowser(PagesBrowser):
Romain Bignon's avatar
Romain Bignon committed
268 269
        # ...

270 271 272 273
        home = URL('/$', IndexPage)
        accounts = URL('/accounts$', ListPage)

Easy, isn't it? The first parameters are regexps of the urls (if you give only a path, it uses the ``BASEURL`` class attribute), and the last one is the class used to handle the response.
Romain Bignon's avatar
Romain Bignon committed
274

275 276 277 278 279
.. note::

    You can handle parameters in the URL using ``(?P<someName>)``. You can then use a keyword argument `someName` to
    bind a value to this parameter in :func:`stay_or_go() <weboob.browser.url.URL.stay_or_go>`.

280
Each time you will go on the home page, ``IndexPage`` will be instanced and set as the ``page`` attribute.
Romain Bignon's avatar
Romain Bignon committed
281

282
For example, we can now implement some methods in ``ExampleBrowser``::
Romain Bignon's avatar
Romain Bignon committed
283

284 285 286
    from weboob.browser import PagesBrowser

    class ExampleBrowser(PagesBrowser):
Romain Bignon's avatar
Romain Bignon committed
287
        # ...
288 289 290 291
        def go_home(self):
            self.home.go()

            assert self.home.is_here()
Romain Bignon's avatar
Romain Bignon committed
292

293 294
        def iter_accounts_list(self):
            self.accounts.stay_or_go()
Romain Bignon's avatar
Romain Bignon committed
295

296
            return self.page.iter_accounts()
Romain Bignon's avatar
Romain Bignon committed
297

298
When calling the :func:`go() <weboob.browser.url.URL.go>` method, it reads the first regexp url of our :class:`URL <weboob.browser.url.URL>` object, and go on the page.
Romain Bignon's avatar
Romain Bignon committed
299

300
:func:`stay_or_go() <weboob.browser.url.URL.stay_or_go>` is used when you want to relocate on the page only if we aren't already on it.
301 302

Once we are on the ``ListPage``, we can call every methods of the ``page`` object.
Romain Bignon's avatar
Romain Bignon committed
303 304 305 306

Use it in backend
-----------------

307
Now you have a functional browser, you can use it in your class ``ExampleModule`` by defining it with the ``BROWSER`` attribute::
Romain Bignon's avatar
Romain Bignon committed
308

309 310 311
    from weboob.tools.backend import Module
    from weboob.capabilities.bank import CapBank

Romain Bignon's avatar
Romain Bignon committed
312 313 314
    from .browser import ExampleBrowser

    # ...
315
    class ExampleModule(Module, CapBank):
Romain Bignon's avatar
Romain Bignon committed
316 317 318
        # ...
        BROWSER = ExampleBrowser

319
You can now access it with member ``browser``. The class is instanced at the first call to this attribute.
Romain Bignon's avatar
Romain Bignon committed
320

321
For example, we can now implement :func:`CapBank.iter_accounts <weboob.capabilities.bank.CapBank.iter_accounts>`::
Romain Bignon's avatar
Romain Bignon committed
322

323 324
    def iter_accounts(self):
        return self.browser.iter_accounts_list()
Romain Bignon's avatar
Romain Bignon committed
325

326
For this method, we only call immediately ``ExampleBrowser.iter_accounts_list``, as there isn't anything else to do around.
Romain Bignon's avatar
Romain Bignon committed
327 328 329 330 331

Login management
----------------

When the website requires to be authenticated, you have to give credentials to the constructor of the browser. You can redefine
332
the method :func:`create_default_browser <weboob.tools.backend.Module.create_default_browser>`::
Romain Bignon's avatar
Romain Bignon committed
333

334 335 336
    from weboob.tools.backend import Module
    from weboob.capabilities.bank import CapBank

337
    class ExampleModule(Module, CapBank):
Romain Bignon's avatar
Romain Bignon committed
338 339 340 341
        # ...
        def create_default_browser(self):
            return self.create_browser(self.config['username'].get(), self.config['password'].get())

342 343
On the browser side, you need to inherit from :func:`LoginBrowser <weboob.browser.browsers.LoginBrowser>` and to implement the function
:func:`do_login <weboob.browser.browsers.LoginBrowser.do_login>`::
Romain Bignon's avatar
Romain Bignon committed
344

345 346 347
    from weboob.browser import LoginBrowser
    from weboob.exceptions import BrowserIncorrectPassword

348 349
    class ExampleBrowser(LoginBrowser):
        login = URL('/login', LoginPage)
Romain Bignon's avatar
Romain Bignon committed
350 351
        # ...

352 353
        def do_login(self):
            self.login.stay_or_go()
Romain Bignon's avatar
Romain Bignon committed
354 355 356

            self.page.login(self.username, self.password)

357 358
            if self.login_error.is_here():
                raise BrowserIncorrectPassword(self.page.get_error())
Romain Bignon's avatar
Romain Bignon committed
359

360 361
You may provide a custom :func:`do_logout <weboob.browser.browsers.LoginBrowser.do_logout>`:: function if you need to customize the default logout process, which simply clears all cookies.

362
Also, your ``LoginPage`` may look like::
Romain Bignon's avatar
Romain Bignon committed
363

364 365
    from weboob.browser.pages import HTMLPage

366 367 368 369 370 371
    class LoginPage(HTMLPage):
        def login(self, username, password):
            form = self.get_form(name='auth')
            form['username'] = username
            form['password'] = password
            form.submit()
Romain Bignon's avatar
Romain Bignon committed
372

373
Then, each method on your browser which needs your user to be authenticated may be decorated by :func:`need_login <weboob.browser.browsers.need_login>`::
Romain Bignon's avatar
Romain Bignon committed
374

375 376 377
    from weboob.browser import LoginBrowser, URL
    from weboob.browser import need_login

378 379
    class ExampleBrowser(LoginBrowser):
        accounts = URL('/accounts$', ListPage)
Romain Bignon's avatar
Romain Bignon committed
380

381 382 383 384
        @need_login
        def iter_accounts(self):
            self.accounts.stay_or_go()
            return self.page.get_accounts()
Romain Bignon's avatar
Romain Bignon committed
385

386 387 388 389 390 391 392 393
You finally have to set correctly the :func:`logged <weboob.browser.pages.Page.logged>` attribute of each page you use.  The
:func:`need_login <weboob.browser.browsers.need_login>` decorator checks if the current page is a logged one by reading the attribute
:func:`logged <weboob.browser.pages.Page.logged>` of the instance. This attributes defaults to  ``False``, which means that :func:`need_login
<weboob.browser.browsers.need_login>` will first call :func:`do_logout <weboob.browser.browsers.LoginBrowser.do_logout>` before calling the
decorated method.

You can either define it yourself, as a class boolean attribute or as a property, or inherit your class from :class:`LoggedPage <weboob.browser.pages.LoggedPage>`.
In the latter case, remember that Python inheritance requires the :class:`LoggedPage <weboob.browser.pages.LoggedPage>` to be placed first such as in::
394
    from weboob.browser.pages import LoggedPage, HTMLPage
395 396 397

    class OnlyForLoggedUserPage(LoggedPage, HTMLPage):
        # ...
Romain Bignon's avatar
Romain Bignon committed
398 399


400 401
Parsing of pages
****************
Romain Bignon's avatar
Romain Bignon committed
402

403
.. note::
404 405
    Depending of the base class you use for your page, it will parse html, json, csv, etc. In this section, we will
    describe the case of HTML documents.
Romain Bignon's avatar
Romain Bignon committed
406 407


408
When your browser locates on a page, an instance of the class related to the
409
:class:`URL <weboob.browser.url.URL>` attribute which matches the url
410 411
is created. You can declare methods on your class to allow your browser to
interact with it.
Romain Bignon's avatar
Romain Bignon committed
412

413 414
The first thing to know is that page parsing is done in a descriptive way. You
don't have to loop on HTML elements to construct the object. Just describe how
415
to get correct data to construct it. It is the ``Browser`` class work to actually
416
construct the object.
Romain Bignon's avatar
Romain Bignon committed
417 418 419

For example::

420
    from weboob.browser.pages import LoggedPage, HTMLPage
421 422
    from weboob.browser.filters.html import Attr
    from weboob.browser.filters.standard import CleanDecimal, CleanText
423
    from weboob.capabilities.bank import Account
424
    from weboob.browser.elements import method, ListElement, ItemElement
425 426

    class ListPage(LoggedPage, HTMLPage):
427 428 429
        @method
        class get_accounts(ListElement):
            item_xpath = '//ul[@id="list"]/li'
Romain Bignon's avatar
Romain Bignon committed
430

431
            class item(ItemElement):
432
                klass = Account
Romain Bignon's avatar
Romain Bignon committed
433

434 435 436
                obj_id = Attr('id')
                obj_label = CleanText('./td[@class="name"]')
                obj_balance = CleanDecimal('./td[@class="balance"]')
437

438 439 440 441
As you can see, we first set ``item_xpath`` which is the xpath string used to iterate over elements to access data. In a
second time we define ``klass`` which is the real class of our object. And then we describe how to fill each object's
attribute using what we call filters. To set an attribute `foobar` of the object, we should fill `obj_foobar`. It can
either be a filter, a constant or a function.
442 443 444 445 446 447 448

Some example of filters:

* :class:`Attr <weboob.browser.filters.html.Attr>`: extract a tag attribute
* :class:`CleanText <weboob.browser.filters.standard.CleanText>`: get a cleaned text from an element
* :class:`CleanDecimal <weboob.browser.filters.standard.CleanDecimal>`: get a cleaned Decimal value from an element
* :class:`Date <weboob.browser.filters.standard.Date>`: read common date formats
449 450 451 452 453 454
* :class:`DateTime <weboob.browser.filters.standard.Date>`: read common datetime formats
* :class:`Env <weboob.browser.filters.standard.Env>`: typically useful to get a named parameter in the URL (passed as a
  keyword argument to :func:`stay_or_go() <weboob.browser.url.URL.stay_or_go>`)
* :class:`Eval <weboob.browser.filters.standard.Eval>`: evaluate a lambda on the given value
* :class:`Format <weboob.browser.filters.standard.Format>`: a formatting filter, uses the standard Python format string
  notations.
455 456 457 458 459
* :class:`Link <weboob.browser.filters.html.Link>`: get the link uri of an element
* :class:`Regexp <weboob.browser.filters.standard.Regexp>`: apply a regex
* :class:`Time <weboob.browser.filters.standard.Time>`: read common time formats
* :class:`Type <weboob.browser.filters.standard.Type>`: get a cleaned value of any type from an element text

460 461
The full list of filters can be found in :doc:`weboob.browser.filters </api/browser/filters/index>`.

462 463 464 465 466 467 468 469 470
Filters can be combined. For example::

    obj_id = Link('./a[1]') & Regexp(r'id=(\d+)') & Type(type=int)

This code do several things, in order:

#) extract the href attribute of our item first ``a`` tag child
#) apply a regex to extract a value
#) convert this value to int type
Romain Bignon's avatar
Romain Bignon committed
471

472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488

When you want to access some attributes of your :class:`HTMLPage <weboob.browser.pages.HTMLPage>` object to fill an
attribute in a Filter, you should use the function construction for this attribute. For example::

	def obj_url(self):
		return (
			u'%s%s' % (
				self.page.browser.BASEURL,
				Link(
					u'//a[1]'
				)(self)
			)
	)

which will return a full URL, concatenating the ``BASEURL`` from the browser
with the (relative) link uri of the first ``a`` tag child.

Romain Bignon's avatar
Romain Bignon committed
489 490 491 492 493 494
.. note::

   All objects ID must be unique, and useful to get more information later

Your module is now functional and you can use this command::

495
    $ boobank -b example list
Romain Bignon's avatar
Romain Bignon committed
496

497 498 499 500 501 502 503
.. note::

	You can pass ``-a`` command-line argument to any Weboob application to log
	all the possible debug output (including requests and their parameters, raw
	responses and loaded HTML pages) in a temporary directory, indicated at the
	launch of the program.

Romain Bignon's avatar
Romain Bignon committed
504 505 506 507 508 509
Tests
*****

Every modules must have a tests suite to detect when there are changes on websites, or when a commit
breaks the behavior of the module.

510
Edit ``test.py`` and write, for example::
Romain Bignon's avatar
Romain Bignon committed
511 512 513 514

    # -*- coding: utf-8 -*-
    from weboob.tools.test import BackendTest

515
    __all__ = ['ExampleTest']
Romain Bignon's avatar
Romain Bignon committed
516 517

    class ExampleTest(BackendTest):
Florent Fourcot's avatar
Florent Fourcot committed
518
        MODULE = 'example'
Romain Bignon's avatar
Romain Bignon committed
519

520 521
        def test_iter_accounts(self):
            accounts = list(self.backend.iter_accounts())
Romain Bignon's avatar
Romain Bignon committed
522

523
            self.assertTrue(len(accounts) > 0)
Romain Bignon's avatar
Romain Bignon committed
524 525 526 527 528

To try running test of your module, launch::

    $ tools/run_tests.sh example

Laurent Bachelier's avatar
Laurent Bachelier committed
529
For more information, look at the :doc:`tests` guides.
530

Romain Bignon's avatar
Romain Bignon committed
531 532 533 534 535 536
Advanced topics
***************

Filling objects
---------------

537 538 539 540 541 542
.. note::

    Filling objects using ``fillobj`` should be used whenever you need to fill some fields automatically based on data
    fetched from the scraping. If you only want to fill some fields automatically based on some static data, you should
    just inherit the base object class and set these fields.

Romain Bignon's avatar
Romain Bignon committed
543 544
An object returned by a method of a capability can be not fully completed.

545 546
The class :class:`Module <weboob.tools.backend.Module>` provides a method named
:func:`fillobj <weboob.tools.backend.Module.fillobj>`, which can be called by an application to
Romain Bignon's avatar
Romain Bignon committed
547 548 549 550
fill some unloaded fields of a specific object, for example with::

    backend.fillobj(video, ['url', 'author'])

551 552 553
The ``fillobj`` method will check on the object which fields (in the ones given in the list argument) are not loaded
(equal to ``NotLoaded``, which is the default value), to reduce the list to the real uncompleted fields, and call the
method associated to the type of the object.
Romain Bignon's avatar
Romain Bignon committed
554 555

To define what objects are supported to be filled, and what method to call, define the ``OBJECTS``
556
class attribute in your ``ExampleModule``::
Romain Bignon's avatar
Romain Bignon committed
557

558 559 560
    from weboob.tools.backend import Module
    from weboob.capabilities.video import CapVideo

561
    class ExampleModule(Module, CapVideo):
562 563 564
        # ...

        OBJECTS = {Video: fill_video}
Romain Bignon's avatar
Romain Bignon committed
565 566 567

The prototype of the function might be::

568
    func(self, obj, fields)
Romain Bignon's avatar
Romain Bignon committed
569 570 571

Then, the function might, for each requested fields, fetch the right data and fill the object. For example::

572 573 574
    from weboob.tools.backend import Module
    from weboob.capabilities.video import CapVideo

575
    class ExampleModule(Module, CapVideo):
576
        # ...
Romain Bignon's avatar
Romain Bignon committed
577

578 579 580
        def fill_video(self, video, fields):
            if 'url' in fields:
                return self.backend.get_video(video.id)
Romain Bignon's avatar
Romain Bignon committed
581

582
            return video
Romain Bignon's avatar
Romain Bignon committed
583

584 585
Here, when the application has got a :class:`Video <weboob.capabilities.video.BaseVideo>` object with
:func:`search_videos <weboob.capabilities.video.CapVideo.search_videos>`, in most cases, there are only some meta-data, but not the direct link to the video media.
Romain Bignon's avatar
Romain Bignon committed
586

587
As our method :func:`get_video <weboob.capabilities.video.CapVideo.get_video>` will get all
Laurent Bachelier's avatar
Laurent Bachelier committed
588
of the missing data, we just call it with the object as parameter to complete it.
589 590 591 592 593 594 595 596 597


Storage
-------

The application can provide a storage to let your backend store data. So, you can define the structure of your storage space::

    STORAGE = {'seen': {}}

598
To store and read data in your storage space, use the ``storage`` attribute of your :class:`Module <weboob.tools.backend.Module>`
599 600 601
object.

It implements the methods of :class:`BackendStorage <weboob.tools.backend.BackendStorage>`.