This documentation is intended for people who wants to contribute or work with the internals of the project, it is not for end-users.
Since version 0.9, Libervia uses SQLAlchemy with its Object–Relational Mapping as a backend to store persistent data, and Alembic is used to handle schema and data migrations.
SQLite is currently the only supported database, but it is planned to add support for other ones (notably PostgreSQL), probably during the development of 0.9 version.
The mapping is done in
sat.memory.sqla_mapping and working with database is done
through high level methods found in
Before the move to SQLAlchemy, there was a strict separation between database
implementation and the rest of the code. With 0.9, objects mapping to database can be used
and manipulated directly outside of
sat.memory.sqla to take profit of SQLAlchemy
Database state is detected when the backend starts, and the database will be created or migrated automatically if necessary.
To create a new migration script,
Alembic may be used directly. To do so, be sure to
have an up-to-date database (and a backup in case of troubles), then activate the virtual
environment where Libervia is installed (Alembic needs to access ORM mapping), go to
sat/memory/migration directory, and enter the following command:
alembic revision --autogenerate -m "some revision message"
This will create a base migration file in
versions directory. Adapt it to your needs,
try to create both
downgrade method whenever possible, and be sure to
test it in both directions (
alembic upgrade head and
<previous_revision>). Please check Alembic documentation for more details.
There is an internal cache for pubsub nodes and items, which is done in
PubsubItem class are the one mapping
The cache is operated transparently to end-user, when a pubsub request is done, it uses a trigger to check if the requested node is or must be cached, and if possible returns result directly from database, otherwise it lets the normal workflow continue and query the pubsub service.
To save resources, not all nodes are fully cached. When a node is checked, a series of analysers are checked, and the first one matching is used to determine if the node must be synchronised or not.
Analysers can be registered by any plugins using
- PubsubCache.registerAnalyser(analyser: dict) None ¶
Register a new pubsub node analyser
analyser – An analyser is a dictionary which may have the following keys
(keys with a
*are mandatory, at least one of
namespacekeys must be used):
- name (str)*:
a unique name for this analyser. This name will be stored in database to retrieve the analyser when necessary (notably to get the parsing method), thus it is recommended to use a stable name such as the source plugin name instead of a name which may change with standard evolution, such as the feature namespace.
- type (str)*:
indicates what kind of items we are dealing with. Type must be a human readable word, as it may be used in searches. Good types examples are blog or event.
- node (str):
prefix of a node name which may be used to identify its type. Example: urn:xmpp:microblog:0 (a node starting with this name will be identified as blog node).
- namespace (str):
root namespace of items. When analysing a node, the first item will be retrieved. The analyser will be chosen its given namespace match the namespace of the first child element of
- to_sync (bool):
if True, the node must be synchronised in cache. The default False value means that the pubsub service will always be requested.
- parser (callable):
method (which may be sync, a coroutine or a method returning a “Deferred”) to call to parse the
domish.Elementof the item. The result must be dictionary which can be serialised to JSON.
The method must have the following signature:
- parser(client: SatXMPPEntity, item_elt: domish.Element, service: Optional[jid.JID], node: Optional[str]) dict
- match_cb (callable):
method (which may be sync, a coroutine or a method returning a “Deferred”) called when the analyser matches. The method is called with the curreny analyse which is can modify in-place.
The method must have the following signature:
- sat.plugins.plugin_pubsub_cache.match_cb(client: SatXMPPEntity, analyse: dict) None
exceptions.Conflict – a analyser with this name already exists
If no analyser is found,
to_sync is false, or an error happens during the caching,
the node won’t be synchronised and the pubsub service will always be requested.
Specifying an optional parser will store parsed data in addition to the raw XML of the items. This is more space consuming, but may be desired for the following reasons:
the parsing is resource consuming (network call or some CPU intensive operations are done)
it is desirable to do queries on parsed data. Indeed the parsed data are stored in a JSON field and its keys may be queried individually.
The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or an user may be doing a low-level pubsub request.