[Wish-list] Privacy issue
This is rather a wish-list issue... but privacy is an important topic!
This could also be a documentation issue because I found nowhere how to get the behaviour I expect... it might still be there... undocumented.
It seems that the behaviour of boobank (and probably other modules because what I describe comes from the core as far as I could see in the code) is to try to update the local repository if something goes wrong with connecting to a bank.
That is a feature a "Jdownloader user" might find desirable, not to bother trying to figure out what went wrong. Indeed if that is due to a change to a scrapped website, updating the local repository can automagically fix the issue.
But the way it is done and the feature itself is questionable for other use cases.
Issue 1: this is a privacy issue ("call home")!
I am using that inside a Linux script to prepare a nice conkyrc. That auto-update with no option to remove it is a "call home" and that is breaching privacy, especially that the feature is automatic and with no (simple) way to stop it.
Proposition: the "privacy conscious" way would be:
- do NOT auto update by default (you can still suggest it with a message)
- add a flag to the command line so that those wanting the auto-update (it is a nice feature) will still have it should they opt-in. Could be like:
- obviously you can still run without the flag and manually do a
weboob-config updatewhen you think if might help fix a connection issue situation.
Since I don't want (yet) to fiddle with the code... especially the core, we read the code (with a colleague) and figured out a simple workaround (seems to work so far).
When there is an error detected, the way to make the core auto-update seems to be to delete a file in ~/.local/share/weboob/repository
There is a single file there (currently: 00-http__updates_weboob_org_1_2_main) that seem to hold the list of modules. The workaround is then to prevent weboob from deleting this file. This is simply done with:
chmod 555 ~/.local/share/weboob/repository
Issue 2: the way the feature is done breaks things!
We have a test machine that is (almost) completely isolated from the local network to avoid unwanted interactions. This machine does not even have internet access by default (due to the strict firewall isolation), unless you plug in a phone with tethering.
Steps to reproduce: to reproduce the bug, do the following steps:
- Configure boobank with any backend
- Disconnect all interfaces that have internet access on your local machine
- input the command
list(enter any ID/Password if requested, since we have no internet that does not matter it will fail anyway!)
- see that you have exceptions followed by a stacktrace (indeed when you have no internet connection, even the DNS fail)
- Remark: this exception could be handled more gracefully, instead of spitting out a stracktrace... but that is not the point of this bug report!
- Now launch boobank again: since the repository configuration file has been deleted it will refuse to start, although everything is still there and working fine!
So, if at some point you happen to have no connection to the internet (intentionally as above or because your connection is really down) weboob auto-destroys itself...
Indeed, since there is the non-removable-auto-update feature, the next time you run it with a working internet connection it will auto-repare itself. But hey, isn't that useless bandwith and I/O, poor performance, whereas letting the user in charge of what he wants to do would have been better?
Final thoughts: I am not yet attaching any proposed code to that (I'm not currently a world class python expert anyway!)... I will just use my chmod 555 workaround for the moment!
Before jumping to code, you might want to discuss:
- if you care more about privacy/letting the user in control, than about things happening automatically
- if you decide to give more control, decide how that should be conveyed to weboob: options on command line, environment variables, core configuration file, all of them, other solution, etc...
- possibly also decide to catch more gracefully some network error exceptions.
- there are already some options available, as
boobank --helpshows, so adding --auto-update (opt in) should respect how things work today.
Issue 1, here is a patch for
console.py(stable version 1.2)
Note: I didn't program the "auto-update" option. I just removed the auto-update feature and instead, included a hint on how to update the modules. As for privacy, this is "good enough", the auto-update feature in console is probably anyway overkill! Since you are in console mode, you shouldn't be afraid of opening another terminal to try an update. If you do so, you do that willingly and there is no more privacy issue.
Should you decide the auto-update here is absolutely needed, and you want to add an option, the code is still there commented. Todo then would be: add the option, test it and do the correct branch: existing code or hint (new).
$ diff ../.local/lib/python2.7/site-packages/weboob-1.2-py2.7.egg/weboob/tools/application/console.py-orig ../.local/lib/python2.7/site-packages/weboob-1.2-py2.7.egg/weboob/tools/application/console.py 602,610c602,613 < self.weboob.repositories.update_repositories(ConsoleProgress(self)) < < # minfo of the new available module < minfo = self.weboob.repositories.get_module_info(backend.NAME) < if minfo and minfo.version > self.weboob.repositories.versions.get(minfo.name) and \ < self.ask('A new version of %s is available. Do you want to install it?' % minfo.name, default=True) and \ < self.install_module(minfo): < print('New version of module %s has been installed. Retry to call the command.' % minfo.name) < return --- > # self.weboob.repositories.update_repositories(ConsoleProgress(self)) > # > # # minfo of the new available module > # minfo = self.weboob.repositories.get_module_info(backend.NAME) > # if minfo and minfo.version > self.weboob.repositories.versions.get(minfo.name) and \ > # self.ask('A new version of %s is available. Do you want to install it?' % minfo.name, default=True) and \ > # self.install_module(minfo): > # print('New version of module %s has been installed. Retry to call the command.' % minfo.name) > # return > print(u'Unexpected behaviour of the website.') > print(u'Please run "weboob-config update" to refresh the module %s.' % minfo.name) > print(u'If it has been updated, you can retry the command.')
Issue 2: in
repositories.py(core) line 576 (stable verion 1.2) of class Repositories / Method update_repositories, does not catch the exception when deleting the repository file is not possible.
Exception should be caught here, file locked could mean we don't want the update. So the message in the catch should say that to update we need to allow deleting files in the repositories directory.Edited
Mentioned in merge request !35