Developer’s guide¶
For a quick start, skip the details below and jump right away to the Cartridge getting started guide.
For a deep dive into what you can develop with Tarantool Cartridge, go on with the Cartridge developer’s guide.
Introduction¶
To develop and start an application, in short, you need to go through the following steps:
Install Tarantool Cartridge and other components of the development environment.
Develop the application. In case it is a cluster-aware application, implement its logic in a custom (user-defined) cluster role to initialize the database in a cluster environment.
Deploy the application to target server(s). This includes configuring and starting the instance(s).
In case it is a cluster-aware application, deploy the cluster.
The following sections provide details for each of these steps.
Installing Tarantool Cartridge¶
Creating a project¶
To set up your development environment, create a project using the Tarantool Cartridge project template. In any directory, say:
$ cartridge create --name <app_name> /path/to/
This will automatically set up a Git repository in a new /path/to/<app_name>/
directory, tag it with version 0.1.0
,
and put the necessary files into it.
In this Git repository, you can develop the application (by simply editing the default files provided by the template), plug the necessary modules, and then easily pack everything to deploy on your server(s).
The project template creates the <app_name>/
directory with the following
contents:
<app_name>-scm-1.rockspec
file where you can specify the application dependencies.deps.sh
script that resolves dependencies from the.rockspec
file.init.lua
file which is the entry point for your application..git
file necessary for a Git repository..gitignore
file to ignore the unnecessary files.env.lua
file that sets common rock paths so that the application can be started from any directory.custom-role.lua
file that is a placeholder for a custom (user-defined) cluster role.
The entry point file (init.lua
), among other things, loads the cartridge
module and calls its initialization function:
...
local cartridge = require('cartridge')
...
cartridge.cfg({
-- cartridge options example
workdir = '/var/lib/tarantool/app',
advertise_uri = 'localhost:3301',
cluster_cookie = 'super-cluster-cookie',
...
}, {
-- box options example
memtx_memory = 1000000000,
... })
...
The cartridge.cfg()
call renders the instance operable via the administrative
console but does not call box.cfg()
to configure instances.
Warning
Calling the box.cfg()
function is forbidden.
The cluster itself will do it for you when it is time to:
bootstrap the current instance once you:
run
cartridge.bootstrap()
via the administrative console, orclick Create in the web interface;
join the instance to an existing cluster once you:
run
cartridge.join_server({uri = 'other_instance_uri'})
via the console, orclick Join (an existing replica set) or Create (a new replica set) in the web interface.
Notice that you can specify a cookie for the cluster (cluster_cookie
parameter)
if you need to run several clusters in the same network. The cookie can be any
string value.
Now you can develop an application that will run on a single or multiple independent Tarantool instances (e.g. acting as a proxy to third-party databases) – or will run in a cluster.
If you plan to develop a cluster-aware application, first familiarize yourself with the notion of cluster roles.
Cluster roles¶
Cluster roles are Lua modules that implement some specific functions and/or logic. In other words, a Tarantool Cartridge cluster segregates instance functionality in a role-based way.
Since all instances running cluster applications use the same source code and are aware of all the defined roles (and plugged modules), you can dynamically enable and disable multiple different roles without restarts, even during cluster operation.
Note that every instance in a replica set performs the same roles and you cannot enable/disable roles individually on some instances. In other words, configuration of enabled roles is set up per replica set. See a step-by-step configuration example in this guide.
Built-in roles¶
The cartridge
module comes with two built-in roles that implement
automatic sharding:
vshard-router
that handles thevshard
’s compute-intensive workload: routes requests to storage nodes.vshard-storage
that handles thevshard
’s transaction-intensive workload: stores and manages a subset of a dataset.Note
For more information on sharding, see the vshard module documentation.
With the built-in and custom roles, you can develop applications with separated compute and transaction handling – and enable relevant workload-specific roles on different instances running on physical servers with workload-dedicated hardware.
Custom roles¶
You can implement custom roles for any purposes, for example:
define stored procedures;
implement extra features on top of
vshard
;go without
vshard
at all;implement one or multiple supplementary services such as e-mail notifier, replicator, etc.
To implement a custom cluster role, do the following:
Take the
app/roles/custom.lua
file in your project as a sample. Rename this file as you wish, e.g.app/roles/custom-role.lua
, and implement the role’s logic. For example:-- Implement a custom role in app/roles/custom-role.lua #!/usr/bin/env tarantool local role_name = 'custom-role' local function init() ... end local function stop() ... end return { role_name = role_name, init = init, stop = stop, }
Here the
role_name
value may differ from the module name passed to thecartridge.cfg()
function. If therole_name
variable is not specified, the module name is the default value.Note
Role names must be unique as it is impossible to register multiple roles with the same name.
Register the new role in the cluster by modifying the
cartridge.cfg()
call in theinit.lua
entry point file:-- Register a custom role in init.lua ... local cartridge = require('cartridge') ... cartridge.cfg({ workdir = ..., advertise_uri = ..., roles = {'custom-role'}, }) ...
where
custom-role
is the name of the Lua module to be loaded.The role module does not have required functions, but the cluster may execute the following ones during the role’s life cycle:
init()
is the role’s initialization function.Inside the function’s body you can call any box functions: create spaces, indexes, grant permissions, etc. Here is what the initialization function may look like:
local function init(opts) -- The cluster passes an 'opts' Lua table containing an 'is_master' flag. if opts.is_master then local customer = box.schema.space.create('customer', { if_not_exists = true } ) customer:format({ {'customer_id', 'unsigned'}, {'bucket_id', 'unsigned'}, {'name', 'string'}, }) customer:create_index('customer_id', { parts = {'customer_id'}, if_not_exists = true, }) end end
Note
Neither
vshard-router
norvshard-storage
manage spaces, indexes, or formats. You should do it within a custom role: add abox.schema.space.create()
call to your first cluster role, as shown in the example above.The function’s body is wrapped in a conditional statement that lets you call
box
functions on masters only. This protects against replication collisions as data propagates to replicas automatically.
stop()
is the role’s termination function. Implement it if initialization starts a fiber that has to be stopped or does any job that needs to be undone on termination.validate_config()
andapply_config()
are functions that validate and apply the role’s configuration. Implement them if some configuration data needs to be stored cluster-wide.Next, get a grip on the role’s life cycle to implement the functions you need.
Defining role dependencies¶
You can instruct the cluster to apply some other roles if your custom role is enabled.
For example:
-- Role dependencies defined in app/roles/custom-role.lua local role_name = 'custom-role' ... return { role_name = role_name, dependencies = {'cartridge.roles.vshard-router'}, ... }
Here
vshard-router
role will be initialized automatically for every instance withcustom-role
enabled.Using multiple vshard storage groups¶
Replica sets with
vshard-storage
roles can belong to different groups. For example,hot
orcold
groups meant to independently process hot and cold data.Groups are specified in the cluster’s configuration:
-- Specify groups in init.lua cartridge.cfg({ vshard_groups = {'hot', 'cold'}, ... })
If no groups are specified, the cluster assumes that all replica sets belong to the
default
group.With multiple groups enabled, every replica set with a
vshard-storage
role enabled must be assigned to a particular group. The assignment can never be changed.Another limitation is that you cannot add groups dynamically (this will become available in future).
Finally, mind the syntax for router access. Every instance with a
vshard-router
role enabled initializes multiple routers. All of them are accessible through the role:local router_role = cartridge.service_get('vshard-router') router_role.get('hot'):call(...)
If you have no roles specified, you can access a static router as before (when Tarantool Cartridge was unaware of groups):
local vhsard = require('vshard') vshard.router.call(...)
However, when using the current group-aware API, you must call a static router with a colon:
local router_role = cartridge.service_get('vshard-router') local default_router = router_role.get() -- or router_role.get('default') default_router:call(...)
Role’s life cycle (and the order of function execution)¶
The cluster displays the names of all custom roles along with the built-in
vshard-*
roles in the web interface. Cluster administrators can enable and disable them for particular instances – either via the web interface or via the cluster public API. For example:cartridge.admin.edit_replicaset('replicaset-uuid', {roles = {'vshard-router', 'custom-role'}})
If you enable multiple roles on an instance at the same time, the cluster first initializes the built-in roles (if any) and then the custom ones (if any) in the order the latter were listed in
cartridge.cfg()
.If a custom role has dependent roles, the dependencies are registered and validated first, prior to the role itself.
The cluster calls the role’s functions in the following circumstances:
The
init()
function, typically, once: either when the role is enabled by the administrator or at the instance restart. Enabling a role once is normally enough.The
stop()
function – only when the administrator disables the role, not on instance termination.The
validate_config()
function, first, before the automaticbox.cfg()
call (database initialization), then – upon every configuration update.The
apply_config()
function upon every configuration update.
As a tryout, let’s task the cluster with some actions and see the order of executing the role’s functions:
Join an instance or create a replica set, both with an enabled role:
validate_config()
init()
apply_config()
Restart an instance with an enabled role:
validate_config()
init()
apply_config()
Disable role:
stop()
.Upon the
cartridge.confapplier.patch_clusterwide()
call:validate_config()
apply_config()
Upon a triggered failover:
validate_config()
apply_config()
Considering the described behavior:
The
init()
function may:Call
box
functions.Start a fiber and, in this case, the
stop()
function should take care of the fiber’s termination.Configure the built-in HTTP server.
Execute any code related to the role’s initialization.
The
stop()
functions must undo any job that needs to be undone on role’s termination.The
validate_config()
function must validate any configuration change.The
apply_config()
function may execute any code related to a configuration change, e.g., take care of anexpirationd
fiber.
The validation and application functions together allow you to change the cluster-wide configuration as described in the next section.
Configuring custom roles¶
You can:
Store configurations for your custom roles as sections in cluster-wide configuration, for example:
# in YAML configuration file my_role: notify_url: "https://localhost:8080"
-- in init.lua file local notify_url = 'http://localhost' function my_role.apply_config(conf, opts) local conf = conf['my_role'] or {} notify_url = conf.notify_url or 'default' end
Download and upload cluster-wide configuration using the web interface or API (via GET/PUT queries to
admin/config
endpoint likecurl localhost:8081/admin/config
andcurl -X PUT -d "{'my_parameter': 'value'}" localhost:8081/admin/config
).Utilize it in your role’s
apply_config()
function.
Every instance in the cluster stores a copy of the configuration file in its working directory (configured by
cartridge.cfg({workdir = ...})
):/var/lib/tarantool/<instance_name>/config.yml
for instances deployed from RPM packages and managed bysystemd
./home/<username>/tarantool_state/var/lib/tarantool/config.yml
for instances deployed from tar+gz archives.
The cluster’s configuration is a Lua table, downloaded and uploaded as YAML. If some application-specific configuration data, e.g. a database schema as defined by DDL (data definition language), needs to be stored on every instance in the cluster, you can implement your own API by adding a custom section to the table. The cluster will help you spread it safely across all instances.
Such section goes in the same file with topology-specific and
vshard
-specific sections that the cluster generates automatically. Unlike the generated, the custom section’s modification, validation, and application logic has to be defined.The common way is to define two functions:
validate_config(conf_new, conf_old)
to validate changes made in the new configuration (conf_new
) versus the old configuration (conf_old
).apply_config(conf, opts)
to execute any code related to a configuration change. As input, this function takes the configuration to apply (conf
, which is actually the new configuration that you validated earlier withvalidate_config()
) and options (theopts
argument that includesis_master
, a Boolean flag described later).
Important
The
validate_config()
function must detect all configuration problems that may lead toapply_config()
errors. For more information, see the next section.When implementing validation and application functions that call
box
ones for some reason, mind the following precautions:Due to the role’s life cycle, the cluster does not guarantee an automatic
box.cfg()
call prior to callingvalidate_config()
.If the validation function calls any
box
functions (e.g., to check a format), make sure the calls are wrapped in a protective conditional statement that checks ifbox.cfg()
has already happened:-- Inside the validate_config() function: if type(box.cfg) == 'table' then -- Here you can call box functions end
Unlike the validation function,
apply_config()
can callbox
functions freely as the cluster applies custom configuration after the automaticbox.cfg()
call.However, creating spaces, users, etc., can cause replication collisions when performed on both master and replica instances simultaneously. The appropriate way is to call such
box
functions on masters only and let the changes propagate to replicas automatically.Upon the
apply_config(conf, opts)
execution, the cluster passes anis_master
flag in theopts
table which you can use to wrap collision-inducingbox
functions in a protective conditional statement:-- Inside the apply_config() function: if opts.is_master then -- Here you can call box functions end
Custom configuration example¶
Consider the following code as part of the role’s module (
custom-role.lua
) implementation:#!/usr/bin/env tarantool -- Custom role implementation local cartridge = require('cartridge') local role_name = 'custom-role' -- Modify the config by implementing some setter (an alternative to HTTP PUT) local function set_secret(secret) local custom_role_cfg = cartridge.confapplier.get_deepcopy(role_name) or {} custom_role_cfg.secret = secret cartridge.confapplier.patch_clusterwide({ [role_name] = custom_role_cfg, }) end -- Validate local function validate_config(cfg) local custom_role_cfg = cfg[role_name] or {} if custom_role_cfg.secret ~= nil then assert(type(custom_role_cfg.secret) == 'string', 'custom-role.secret must be a string') end return true end -- Apply local function apply_config(cfg) local custom_role_cfg = cfg[role_name] or {} local secret = custom_role_cfg.secret or 'default-secret' -- Make use of it end return { role_name = role_name, set_secret = set_secret, validate_config = validate_config, apply_config = apply_config, }
Once the configuration is customized, do one of the following:
continue developing your application and pay attention to its versioning;
(optional) enable authorization in the web interface.
in case the cluster is already deployed, apply the configuration cluster-wide.
Applying custom role’s configuration¶
With the implementation showed by the example, you can call the
set_secret()
function to apply the new configuration via the administrative console – or an HTTP endpoint if the role exports one.The
set_secret()
function callscartridge.confapplier.patch_clusterwide()
which performs a two-phase commit:It patches the active configuration in memory: copies the table and replaces the
"custom-role"
section in the copy with the one given by theset_secret()
function.The cluster checks if the new configuration can be applied on all instances except disabled and expelled. All instances subject to update must be healthy and
alive
according to the membership module.(Preparation phase) The cluster propagates the patched configuration. Every instance validates it with the
validate_config()
function of every registered role. Depending on the validation’s result:If successful (i.e., returns
true
), the instance saves the new configuration to a temporary file namedconfig.prepare.yml
within the working directory.(Abort phase) Otherwise, the instance reports an error and all the other instances roll back the update: remove the file they may have already prepared.
(Commit phase) Upon successful preparation of all instances, the cluster commits the changes. Every instance:
Creates the active configuration’s hard-link.
Atomically replaces the active configuration file with the prepared one. The atomic replacement is indivisible – it can either succeed or fail entirely, never partially.
Calls the
apply_config()
function of every registered role.
If any of these steps fail, an error pops up in the web interface next to the corresponding instance. The cluster does not handle such errors automatically, they require manual repair.
You will avoid the repair if the
validate_config()
function can detect all configuration problems that may lead toapply_config()
errors.Using the built-in HTTP server¶
The cluster launches an
httpd
server instance during initialization (cartridge.cfg()
). You can bind a port to the instance via an environmental variable:-- Get the port from an environmental variable or the default one: local http_port = os.getenv('HTTP_PORT') or '8080' local ok, err = cartridge.cfg({ ... -- Pass the port to the cluster: http_port = http_port, ... })
To make use of the
httpd
instance, access it and configure routes inside theinit()
function of some role, e.g. a role that exposes API over HTTP:local function init(opts) ... -- Get the httpd instance: local httpd = cartridge.service_get('httpd') if httpd ~= nil then -- Configure a route to, for example, metrics: httpd:route({ method = 'GET', path = '/metrics', public = true, }, function(req) return req:render({json = stat.stat()}) end ) end end
For more information on using Tarantool’s HTTP server, see its documentation.