testing¶
You’ll find here the documentation to run tests on Libervia. If you plan to contribute to the ecosystem, you should use them to check that your modification is not breaking anything, and if possible you should extend them to cover any new feature.
overview¶
Tests are run using pytest and are located in the tests
directory.
You’ll also find legacy tests in libervia/backend/test
but those one are old, not maintained and
only kept there temporarily until they are ported to the new system.
For now, emphasis is put on end-2-end tests, as they are covering the whole ecosystem, and are quite easy to write. The downside is that they are quite long to run.
Several fixtures are available in the various conftest.py
files, it’s a good idea
to have an overview of them if you’re willing to write your own tests.
end-to-end tests¶
End-to-end tests are located in tests/e2e
. They are launched in a well defined
environment managed through Docker. The docker/docker-compose-e2e.yml
is used to
create the suitable containers.
A script is available at tests/e2e/run_e2e.py
to launch the tests. It will create the
containers, bind the current code to them, and set environment variables according to
arguments.
The arguments set to this script are the pytest
options, thus you can have a verbose
mode with -v
and select specific test with -k EXPRESSION
(see pytest --help
for
details).
In addition to pytest option, some flags can be set with the following arguments:
--visual
Launch a VNC viewer to see in real time browser based tests. You must have
vncviewer
executable available in your path (this is part of TigerVNC)--keep-containers
Do no remove Docker container after the end of tests.
--keep-profiles
Do not delete test profiles after the end of tests
--keep-vnc
Do not stop VNC viewer after the end of tests. This argument implies
--visual
.--keep-browser
Do not kill the browser inside the container after tests are done. This argument implies
--keep-container
and--keep-vnc
.--dev-mode
Shortcut for
--keep-containers
,--keep-profiles
and--keep-vnc
. This is useful, as you guess with its names, for development of tests. User can then log-in into thebackend
container, launch a Python console, and work with the automated browser in real-time. Basic commands to launch a browser and log-in with test account are printed at the end of the tests. Note that if you want to have profiles created, or extra tools like the fake SMTP server, you’ll have to launch at least one test which require them. To log-in into thebackend
container, you can use the following command, from/docker
directory:$ docker-compose -f docker-compose-e2e.yml exec backend /bin/bash
Then run a python console with given instructions
It’s also good to know that in the e2e test environment, the following pytest plugins are installed and used:
- pytest-timeout
To avoid having test stuck, it’s good to terminate them after a while. A timeout of 60s is set by default for each test (lower value can give false negatives, as some e2e tests can be long, notably with Selenium).
- pytest-dependency
Even if good testing practice normally means that tests can be run independently, in the case of e2e tests we are using a real environment, and some tests do create files, PubSub nodes, accounts, etc. It would be resource consuming to delete then recreate them only to have standalone tests, thus to keep tests short and simple, some of them must be run in order. The dependecy plugin is used to manage that, and will skip tests if one of their dependencies is failing. The markup help also to document the tests order.
common fixtures¶
Here are the fixture common to all e2e tests which are good to know:
test_profiles
Creates a bunch of test accounts which are available during the whole test session. Those account are destroyed once all the tests are finished (successful or not), except if you set the
LIBERVIA_TEST_E2E_KEEP_PROFILES
environment variable (or use the--keep-profiles
flag inrun_e2e.py
.The profiles created are in the form
accountX
for account on theserver1.test
, oraccountX_sY
for account on other servers (see the docstring for details).This fixture should be used on top of each e2e test module.
pubsub_nodes
Create 2 pubsub nodes with
open
access model and namedtest
(one onaccount1
PEP service, and the other one onpubsub.server1.test
, created with the same account).Those node are created for the scope of the class.
fake_file
Create files filled with random bytes, and check them.
A file is created by calling
fake_file.size(size)
, and by default files of the same size are re-used (setuse_cache=False
to create new files). This method returns apathlib.Path
. SHA-256 hash of the created file can be retrieved usingfake_file.get_source_hash(source_file_path)
with the file path as argument.fake_file.new_dest_file()
will return a Path to a randomly named destination file, andfake_file.get_dest_hash(dest_file_path)
will generate its hash once written.sent_emails
When used, a fake SMTP server (already configured in container’s
libervia.conf
) will be launched if it’s not already, and all messages sent to it since the beginning of the test will be available in the given list. Message are subclasses ofemail.message.EmailMessage
with the extra propertiesfrom_
,to
,subject
andbody
available for easy access to their content.The SMTP server is terminated at the end of the test session.
libervia-cli e2e tests¶
End-to-end tests for libervia-cli
are a good way to tests backend features without having to
deal with frontends UI. Those tests use extensively the sh
module, which helps
writing libervia-cli
commands like if they where methods.
Among the helping fixture (check the various conftest.py
files for details), the
following are specially good to know:
li_json
Set the
json_raw
output are parse it. When you use this instead of the normallibervia-cli
, you’ll get a Python object that you can manipulate easily.li_elt
Set the
xml_raw
output and parse it as a Twisteddomish.Element
. When you use a command which can return XML, it is useful to get this object which is easy to manipulate in Python.editor
Create a fake editor (replacing the one normally set in
EDITOR
environment variable), to automatically modify and/or check the text sent by a command. You can specify Python code to execute to modify the received text with theset_filter
method (this code is in a string which will be executed by Python interpreter, where thecontent
variable is the received text). By default, the text is kept unmodified.After
editor
has been used by thelibervia-cli
command, you can check itsoriginal_content
property to see the text that it received, andnew_content
property to see the text that has been written after updating the original content with the code set inset_filter
.
Libervia e2e tests¶
E2e tests for Libervia are executed, as it is common in web world, with Selenium: user actions are simulated in automated browser, and results are checked.
To make the tests as easy to write as possible, and as natural to read as possible, the higher level Helium Python module is used. Thanks to it, the tests can read pretty much like instructions we would give to a human user. Helium makes also easy to do some tasks more complicated with Selenium alone, like dropping a file to an element.
If a test is failing, a screenshot of the browser is taken. If you run the tests though
the run_e2e.py
command (which you should), you’ll find the screenshots in the
report_*
directory which is created in working dir in case of failure.
Here are the helping fixtures which are notably good to know, you should always use either
log_in_account1
or nobody_logged_in
:
log_in_account1
Start the test with the main test account logged.
nobody_logged_in
Start the test without anybody logged (this is done by clearing all cookies).
examples¶
Following examples have to be run from tests/e2e
directory.
Run all tests for Libervia CLI
:
$ ./run_e2e.py -k libervia-cli
Run all tests for Libervia Web
with real-time visual feedback (note that you need to have
vncviewer
installed and available in path, see above):
$ ./run_e2e.py -k libervia-web --visual
Run all tests with verbose mode (useful to know which test is currently running):
$ ./run_e2e.py -v
Run pubsub tests in verbose mode:
$ ./run_e2e.py -k pubsub -v
Run in dev mode, to work on new tests, note that we run the user_can_create_account
test to be sure to have test profiles created and fake SMTP server run…:
$ ./run_e2e.py -k user_can_create_account --dev-mode
…then to go into the backend
container and work with the browser (to be run in docker
directory)…:
$ docker-compose -f docker-compose-e2e.yml exec backend /bin/bash
…and, inside the container, you can now run python3
and enter instruction prints at
the end of the test session.