From 9db00d705606b47ae3d5b487d687df132b75f21b Mon Sep 17 00:00:00 2001 From: "Phyks (Lucas Verney)" Date: Tue, 3 Jan 2017 17:19:27 +0100 Subject: [PATCH] Rewrite Weboob doc guides according to latest changes in CI and modules. --- CONTRIBUTING.md | 52 ++++++++----------- docs/source/guides/contribute.rst | 82 +++++++++--------------------- docs/source/guides/module.rst | 83 +++++++++++++++++++++++++------ docs/source/guides/tests.rst | 4 ++ 4 files changed, 116 insertions(+), 105 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0570e6384f..0cd58892cd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,52 +4,42 @@ How to contribute Write a patch ------------- -Help yourself with the [documentation](http://dev.weboob.org). +Help yourself with the `documentation `_. -Find an [opened issue on gitlab](https://git.weboob.org/weboob/devel/issues), -or write you own bugfix or feature. Then, once it is necessary, commit with: +Find an opened issue on `this website `_, or write your own bugfix or feature. +Then, once it is necessary, commit with:: -``` -$ git commit -a -``` + $ git commit -a Do not forget to write a helpful commit message. Check your patch ---------------- -You can run these scripts to be sure your patch doesn’t break anything: -``` -$ tools/pyflakes.sh -$ tools/run_tests.sh yourmodulename # or without yourmodulename to test everything -``` +You can run these scripts to be sure your patch doesn't break anything:: -Perhaps you should also write or fix tests. + $ tools/pyflakes.sh + $ tools/weboob_lint.sh + $ tools/run_tests.sh yourmodulename # or without yourmodulename to test everything -Send a patch ------------- +Perhaps you should also write or fix tests. These tests are automatically run by +`Gitlab CI `_ at each commit and merge requests. -``` -$ git format-patch -n -s origin -``` +Create a merge request or send a patch +-------------------------------------- -Then, send them with this command: +The easiest way to send your patch is to create a fork on `the Weboob Gitlab `_ and create a merge +request from there. This way, the code review process is easier and continuous integration is run automatically (see +previous section). -``` -$ git send-email --to=weboob@weboob.org *.patch -``` +If you prefer good old email patches, just use -You can also send the files by yourself if you haven’t any configured MTA on -your system. +:: -Create a fork on git.weboob.org -------------------------------- + $ git format-patch -n -s origin -If you think you’ll contribute to Weboob regularly, you can create a repository -on [git.weboob.org](git.weboob.org). To do so, -[signup](https://git.weboob.org/users/sign_in) and click on _Fork_ on the weboob -repository's page. +Then, send them with this command:: -Then, you can push your commits, and ask for merge requests. + $ git send-email --to=weboob@weboob.org *.patch -All git branch are listed here: https://git.weboob.org/explore/projects \ No newline at end of file +You can also send the files by yourself if you haven't any configured MTA on your system. diff --git a/docs/source/guides/contribute.rst b/docs/source/guides/contribute.rst index adb89646a6..0753b80597 100644 --- a/docs/source/guides/contribute.rst +++ b/docs/source/guides/contribute.rst @@ -9,7 +9,8 @@ Write a patch Help yourself with the `documentation `_. -Find an opened issue on this website, or write you own bugfix or feature. Then, once it is necessary, commit with:: +Find an opened issue on `this website `_, or write your own bugfix or feature. +Then, once it is necessary, commit with:: $ git commit -a @@ -21,12 +22,20 @@ Check your patch You can run these scripts to be sure your patch doesn't break anything:: $ tools/pyflakes.sh + $ tools/weboob_lint.sh $ tools/run_tests.sh yourmodulename # or without yourmodulename to test everything -Perhaps you should also write or fix tests. +Perhaps you should also write or fix tests. These tests are automatically run by +`Gitlab CI `_ at each commit and merge requests. -Send a patch ------------- +Create a merge request or send a patch +-------------------------------------- + +The easiest way to send your patch is to create a fork on `the Weboob Gitlab `_ and create a merge +request from there. This way, the code review process is easier and continuous integration is run automatically (see +previous section). + +If you prefer good old email patches, just use :: @@ -38,62 +47,17 @@ Then, send them with this command:: You can also send the files by yourself if you haven't any configured MTA on your system. -Ask for a public repository on git.symlink.me ---------------------------------------------- - -If you think you'll contribute to Weboob regularly, you can ask for a public repository. You'll also be able to push your commits in, and they'll be merged into the main repository easily. - -All git branch are listed here: http://git.symlink.me/ - -By hosting a buildbot slave -*************************** - -To be sure weboob works fine on lot of architectures, OS and configurations, but also that websites haven't changed and backends still support them, it's important to have enough buildbot slaves. - -If you are interested by hosting a buildbot slave, follow these instructions: - -Create a slave --------------- - -Firstly, you have to install ``pyflakes``, ``nose`` and `buildbot `_. - -Run:: - - $ buildslave create-slave buildbot.weboob.org:9080 - -.. note:: - if you use an old version of buildbot, run ``buildbot`` instead of ``buildslave``. - -Parameters are: - -* **dirname** — the path where you want to setup your slave on your host. -* **name** — the name of your slave. It would be for example your name, your nickname, your hostname. Check on http://buildbot.weboob.org the name you want to use isn't already taken. -* **password** — choose a password to login on the master. - -For example:: - - $ buildslave create-slave /home/me/buildbot buildbot.weboob.org:9080 me secret123 - -Then, edit files in ``/home/me/buildbot/info/`` and run the slave:: - - $ buildslave start /home/me/buildbot - -Contact us ----------- - -To connect your slave to our master, you can send us an email on admin@weboob.org with the following information: - -* The name of your slave; -* The IP address of the host; -* The password of your slave; -* Indicate if you want to run tests for every merges (three times a day) or only do a nightly build. - -When your slave will be accepted, you will see it on http://buildbot.weboob.org/waterfall. +By hosting a Gitlab CI runner +***************************** -How it works ------------- +To be sure weboob works fine on lot of architectures, OS and configurations, but also that websites haven't changed and +backends still support them, it's important to have enough runners with different configurations, especially since +running some tests requires a working backend. -When a build is requested by master, your slave updates its local git repository, and run ``tools/run_tests.sh``. +If you are interested by hosting a Gitlab-CI runner, follow these instructions: -To work correctly, we suggest you to add as many as possible backends with the user of the slave. No private information will be sent to master, and it's better to have tests on backends which need authentication, because not every developers have accounts on them. +You can `install a Gitlab runner `_ and make it use a specific backend file (be +it either by creating a dedicated Docker image with your credentials or running it in ``shell`` mode and making the +backend file available to it). +Then, you should contact us at admin@weboob.org so that we could help you register your runner with our Gitlab. diff --git a/docs/source/guides/module.rst b/docs/source/guides/module.rst index 188d1f893c..9c2afb4f85 100644 --- a/docs/source/guides/module.rst +++ b/docs/source/guides/module.rst @@ -31,7 +31,7 @@ For example, there is the :class:`CapMessages ` are: * **label** - human readable description of a value -* **required** - if ``True``, the backend can't loaded if the key isn't found in its configuration +* **required** - if ``True``, the backend can't be loaded if the key isn't found in its configuration * **default** - an optional default value, used when the key is not in config. If there is no default value and the key is not found in configuration, the **required** parameter is implicitly set * **masked** - if ``True``, the value is masked. It is useful for applications to know if this key is a password -* **regexp** - if specified, on load the specified value is checked against this regexp, and an error is raised if it doesn't match +* **regexp** - if specified, the specified value is checked against this regexp upon loading, and an error is raised if + it doesn't match * **choices** - if this parameter is set, the value must be in the list .. note:: @@ -132,6 +139,11 @@ Available parameters of :class:`Value ` are: There is a special class, :class:`ValueBackendPassword `, which is used to manage private parameters of the config (like passwords or sensible information). +.. note:: + + Other classes are available to store specific types of configuration options. See :mod:`weboob.tools.value + ` for a full list of them. + For example:: from weboob.tools.value import Value, ValueBool, ValueInt, ValueBackendPassword @@ -173,14 +185,18 @@ You need to implement each method of all of the capabilities your module impleme If you ran the ``boilerplate`` script command ``cap``, every methods are already in ``module.py`` and documented. Read :class:`documentation of the capability ` to know what are types of arguments, -what are expected returned objects, and what exceptions it may raises. +what are expected returned objects, and what exceptions it may raise. + +When you are done writing your module, you should remove all the not implemented methods from your module, as the base +capability code will anyway ``raise NotImplementedError()``. Browser ******* Most of modules use a class derived from :class:`PagesBrowser ` or -:class:`LoginBrowser ` (for authenticated websites) to interact with a website. +:class:`LoginBrowser ` (for authenticated websites) to interact with a website or +:class:`APIBrowser ` to interact with an API. Edit ``browser.py``:: @@ -242,6 +258,11 @@ Then, you have to declare them in your browser, with the :class:`URL )``. You can then use a keyword argument `someName` to + bind a value to this parameter in :func:`stay_or_go() `. + Each time you will go on the home page, ``IndexPage`` will be instanced and set as the ``page`` attribute. For example, we can now implement some methods in ``ExampleBrowser``:: @@ -278,7 +299,7 @@ Now you have a functional browser, you can use it in your class ``ExampleModule` You can now access it with member ``browser``. The class is instanced at the first call to this attribute. -For example, we can now implement :func:`CapBank.iter_accounts `:: def iter_accounts(self): return self.browser.iter_accounts_list() @@ -339,7 +360,8 @@ Parsing of pages **************** .. note:: - Depending of the base class you use for your page, it will parse html, json, csv, etc. In our case, it will be only html documents. + Depending of the base class you use for your page, it will parse html, json, csv, etc. In this section, we will + describe the case of HTML documents. When your browser locates on a page, an instance of the class related to the @@ -349,7 +371,7 @@ interact with it. The first thing to know is that page parsing is done in a descriptive way. You don't have to loop on HTML elements to construct the object. Just describe how -to get correct data to construct it. It is the Browser class work to actually +to get correct data to construct it. It is the ``Browser`` class work to actually construct the object. For example:: @@ -370,10 +392,10 @@ For example:: obj_label = CleanText('./td[@class="name"]') obj_balance = CleanDecimal('./td[@class="balance"]') -As you see, we first set ``item_xpath`` which is the xpath string used to -iterate over elements to access data. In a second time we define ``klass`` which -is the real class of our object. And then we describe how to fill each object's -attribute using what we call filters. +As you can see, we first set ``item_xpath`` which is the xpath string used to iterate over elements to access data. In a +second time we define ``klass`` which is the real class of our object. And then we describe how to fill each object's +attribute using what we call filters. To set an attribute `foobar` of the object, we should fill `obj_foobar`. It can +either be a filter, a constant or a function. Some example of filters: @@ -381,11 +403,19 @@ Some example of filters: * :class:`CleanText `: get a cleaned text from an element * :class:`CleanDecimal `: get a cleaned Decimal value from an element * :class:`Date `: read common date formats +* :class:`DateTime `: read common datetime formats +* :class:`Env `: typically useful to get a named parameter in the URL (passed as a + keyword argument to :func:`stay_or_go() `) +* :class:`Eval `: evaluate a lambda on the given value +* :class:`Format `: a formatting filter, uses the standard Python format string + notations. * :class:`Link `: get the link uri of an element * :class:`Regexp `: apply a regex * :class:`Time `: read common time formats * :class:`Type `: get a cleaned value of any type from an element text +The full list of filters can be found in :doc:`weboob.browser.filters `. + Filters can be combined. For example:: obj_id = Link('./a[1]') & Regexp(r'id=(\d+)') & Type(type=int) @@ -396,6 +426,23 @@ This code do several things, in order: #) apply a regex to extract a value #) convert this value to int type + +When you want to access some attributes of your :class:`HTMLPage ` object to fill an +attribute in a Filter, you should use the function construction for this attribute. For example:: + + def obj_url(self): + return ( + u'%s%s' % ( + self.page.browser.BASEURL, + Link( + u'//a[1]' + )(self) + ) + ) + +which will return a full URL, concatenating the ``BASEURL`` from the browser +with the (relative) link uri of the first ``a`` tag child. + .. note:: All objects ID must be unique, and useful to get more information later @@ -437,6 +484,12 @@ Advanced topics Filling objects --------------- +.. note:: + + Filling objects using ``fillobj`` should be used whenever you need to fill some fields automatically based on data + fetched from the scraping. If you only want to fill some fields automatically based on some static data, you should + just inherit the base object class and set these fields. + An object returned by a method of a capability can be not fully completed. The class :class:`Module ` provides a method named @@ -445,9 +498,9 @@ fill some unloaded fields of a specific object, for example with:: backend.fillobj(video, ['url', 'author']) -The ``fillobj`` method will check on the object what fields, in the ones given in list, which are -not loaded (equal to ``NotLoaded``, which is the default value), to reduce the list to the real -uncompleted fields, and call the method associated to the type of the object. +The ``fillobj`` method will check on the object which fields (in the ones given in the list argument) are not loaded +(equal to ``NotLoaded``, which is the default value), to reduce the list to the real uncompleted fields, and call the +method associated to the type of the object. To define what objects are supported to be filled, and what method to call, define the ``OBJECTS`` class attribute in your ``ExampleModule``:: diff --git a/docs/source/guides/tests.rst b/docs/source/guides/tests.rst index ba0cf66487..5c85ab4803 100644 --- a/docs/source/guides/tests.rst +++ b/docs/source/guides/tests.rst @@ -19,6 +19,10 @@ It looks for every files named ``test.py``, and find classes derivated from ``Te Then, it run every method which name starts with ``test_``. +.. note:: + Some environment variables are available, to use specific backend file or send the test results. Refer to `the + comments in the script `_ for more infos on this. + Write a test case ***************** -- GitLab