Developer Documentation

This documentation is intended for people who wants to contribute or work with the internals of the project, it is not for end-users.

Storage

Since version 0.9, Libervia uses SQLAlchemy with its Object–Relational Mapping as a backend to store persistent data, and Alembic is used to handle schema and data migrations.

SQLite is currently the only supported database, but it is planned to add support for other ones (notably PostgreSQL), probably during the development of 0.9 version.

The mapping is done in sat.memory.sqla_mapping and working with database is done through high level methods found in sat.memory.sqla.

Before the move to SQLAlchemy, there was a strict separation between database implementation and the rest of the code. With 0.9, objects mapping to database can be used and manipulated directly outside of sat.memory.sqla to take profit of SQLAlchemy possibilities.

Database state is detected when the backend starts, and the database will be created or migrated automatically if necessary.

To create a new migration script, Alembic may be used directly. To do so, be sure to have an up-to-date database (and a backup in case of troubles), then activate the virtual environment where Libervia is installed (Alembic needs to access ORM mapping), go to sat/memory/migration directory, and enter the following command:

alembic revision --autogenerate -m "some revision message"

This will create a base migration file in versions directory. Adapt it to your needs, try to create both upgrade and downgrade method whenever possible, and be sure to test it in both directions (alembic upgrade head and alembic downgrade <previous_revision>). Please check Alembic documentation for more details.

Pubsub Cache

There is an internal cache for pubsub nodes and items, which is done in plugin_pubsub_cache. The PubsubNode and PubsubItem class are the one mapping the database.

The cache is operated transparently to end-user, when a pubsub request is done, it uses a trigger to check if the requested node is or must be cached, and if possible returns result directly from database, otherwise it lets the normal workflow continue and query the pubsub service.

To save resources, not all nodes are fully cached. When a node is checked, a series of analysers are checked, and the first one matching is used to determine if the node must be synchronised or not.

Analysers can be registered by any plugins using registerAnalyser method:

PubsubCache.registerAnalyser(analyser: dict) → None

Register a new pubsub node analyser

Parameters:

analyser – An analyser is a dictionary which may have the following keys

(keys with a * are mandatory, at least one of node or namespace keys must be used):

name (str)*:

a unique name for this analyser. This name will be stored in database to retrieve the analyser when necessary (notably to get the parsing method), thus it is recommended to use a stable name such as the source plugin name instead of a name which may change with standard evolution, such as the feature namespace.

type (str)*:

indicates what kind of items we are dealing with. Type must be a human readable word, as it may be used in searches. Good types examples are blog or event.

node (str):

prefix of a node name which may be used to identify its type. Example: urn:xmpp:microblog:0 (a node starting with this name will be identified as blog node).

namespace (str):

root namespace of items. When analysing a node, the first item will be retrieved. The analyser will be chosen its given namespace match the namespace of the first child element of <item> element.

to_sync (bool):

if True, the node must be synchronised in cache. The default False value means that the pubsub service will always be requested.

parser (callable):

method (which may be sync, a coroutine or a method returning a “Deferred”) to call to parse the domish.Element of the item. The result must be dictionary which can be serialised to JSON.

The method must have the following signature:

parser(client: SatXMPPEntity, item_elt: domish.Element, service: Optional[jid.JID], node: Optional[str]) → dict

match_cb (callable):

method (which may be sync, a coroutine or a method returning a “Deferred”) called when the analyser matches. The method is called with the curreny analyse which is can modify in-place.

The method must have the following signature:

sat.plugins.plugin_pubsub_cache.match_cb(client: SatXMPPEntity, analyse: dict) → None

Raises:

exceptions.Conflict – a analyser with this name already exists

If no analyser is found, to_sync is false, or an error happens during the caching, the node won’t be synchronised and the pubsub service will always be requested.

Specifying an optional parser will store parsed data in addition to the raw XML of the items. This is more space consuming, but may be desired for the following reasons:

• the parsing is resource consuming (network call or some CPU intensive operations are done)

• it is desirable to do queries on parsed data. Indeed the parsed data are stored in a JSON field and its keys may be queried individually.

The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or an user may be doing a low-level pubsub request.