Module membership

Membership library for Tarantool based on a gossip protocol.

This library builds a mesh from multiple tarantool instances. The mesh monitors itself, helps members discover everyone else and get notified about their status changes with low latency.

It is built upon the ideas from consul, or, more precisely, the SWIM algorithm.

Membership module works over UDP protocol and can operate even before tarantool box.cfg was initialized.

Functions

init (advertise_host, port) Initialize the membership module.
broadcast () Discover members in local network.
leave () Gracefully leave the membership group.
members () Obtain all members known to the current instance.
pairs () Iterate over members.
get_member (uri) Get info about member with the given URI.
myself () Get info about the current instance.
add_member (uri) Add a member to the group.
probe_uri (uri) Send a ping to a member.
set_payload (key, value) Update payload and disseminate it along with the member status.

Tables

MemberInfo Member data structure.

Encryption Functions

get_encryption_key () Retrieve the encryption key that is currently in use.
set_encryption_key (key) Set the key used for low-level message encryption.

Subscription Functions

subscribe () Subscribe for updates in the members table.
unsubscribe (cond) Unsubscribe from membership updates.

membership.options Functions

options.PROTOCOL_PERIOD_SECONDS Period of sending direct PINGs.
options.ACK_TIMEOUT_SECONDS Time to wait for ACK message after PING.
options.ANTI_ENTROPY_PERIOD_SECONDS Period to perform anti-entropy sync.
options.SUSPECT_TIMEOUT_SECONDS Timeout to mark suspect members as dead.
options.NUM_FAILURE_DETECTION_SUBGROUPS Number of members to try indirectly pinging a suspect.
options.MAX_PACKET_SIZE Maximum size of UPD packets to send.
options.ENCRYPTION_INIT Initialization vector for aes256 CBC encryption.


Functions

init (advertise_host, port)
Initialize the membership module. Bind a UDP socket to 0.0.0.0:<port>, set the advertise_uri parameter to <advertise_host>:<port>, and incarnation to 1.

The init() function can be called several times, the old socket will be closed and a new one opened.

If the advertise_uri changes during the next init(), the old URI is considered DEAD. In order to leave the group gracefully use the leave function.

Parameters:

  • advertise_host string either hostname or IP address being advertised to other members
  • port number UDP port to bind and advertise

Returns:

    boolean true

Raises:

Socket bind error
broadcast ()
Discover members in local network. Send UDP broadcast to all networks discovered by getifaddrs() C call

Returns:

    true if broadcast was sent

Or

    false if getifaddrs() fails.
leave ()
Gracefully leave the membership group. The node will be marked with the status left and no other members will ever try to reconnect it.

Returns:

    boolean true if call succeeds, false if member has already left.
members ()
Obtain all members known to the current instance.

Editing this table has no effect.

Returns:

    table a table with URIs as keys and corresponding MemberInfo as values.
pairs ()
Iterate over members. A shorthand for pairs(membership.members()).

Returns:

    Lua iterator

Usage:

    for uri, member in membership.pairs() do end
get_member (uri)
Get info about member with the given URI.

Parameters:

  • uri string <advertise_uri> of member of interest

Returns:

    MemberInfo the member data structure of the instance with the given URI.
myself ()
Get info about the current instance.

Returns:

    MemberInfo the member data structure of the current instance.
add_member (uri)
Add a member to the group. Also propagate this event to other members. Adding a member to a single instance is enough as everybody else in the group will receive the update with time. It does not matter who adds whom.

Warning: The gossip protocol guarantees that every member in the group becomes aware of any status change in two communication cycles.

Parameters:

  • uri string <advertise_uri> of member to add

Returns:

  1. true or nil
  2. optional string

    Possible errors:

    • "parse error" - if the URI can not be parsed
probe_uri (uri)
Send a ping to a member. Send a ping-message to a member to make sure it is in the group.

If the member responds but not in the group, it is added.

If it already is in the group, nothing happens.

Warning: When destination IP can be resolved in several diffent ways (by different hostnames) it is possible that probe_uri() function returns "no response" error, but the member is added to the group with another URI, corresponding to its <advertise_uri>.

Parameters:

  • uri string <advertise_uri> of member to ping

Returns:

  1. true or nil
  2. optional string

    Possible errors:

    • "parse error" - if the URI can not be parsed
    • "ping was not sent" - if hostname could not be reloved
    • "no reponce" - if member does not responf within 0.2 seconds
set_payload (key, value)
Update payload and disseminate it along with the member status. Also increments incarnation.

Parameters:

  • key string a key to set in payload table
  • value auxiliary data

Tables

MemberInfo
Member data structure. A member is represented by the table with the following fields:

Fields:

  • uri string <advertise_uri> of a member
  • status string

    a string that takes one of the values below

    • alive: a member that replies to ping-messages is alive and well.
    • suspect: if any member in the group cannot get a reply from any other member, the first member asks three other alive members to send a ping-message to the member in question. If there is no response, the latter becomes a suspect.
    • dead: a suspect becomes dead after a timeout.
    • left: a member gets the left status after executing the leave function.
  • incarnation number a value incremented every time the instance status changes, or its payload is updated
  • payload table an auxiliary data that can be used by various modules
  • timestamp number a value of fiber.time64() which corresponds to the last update of status or incarnation; it is always local and does not depend on other members’ clock setting.
  • clock_delta number difference of clocks (fiber.time64) between self and peer calculated during ping/ack protocol step or while probe_uri call

Usage:

    tarantool> membership.myself()
---
uri: "localhost:33001"
status: "alive"
incarnation: 1
payload:
    uuid: "2d00c500-2570-4019-bfcc-ab25e5096b73"
timestamp: 1522427330993752
clock_delta: 700
...

Encryption Functions

The encryption is handled by the crypto.cipher.aes256.cbc Tarantool module.

For proper communication, all members must be configured to use the same encryption key. Otherwise, members report either dead or non-decryptable in their status.

get_encryption_key ()
Retrieve the encryption key that is currently in use.

Returns:

    string encryption key
set_encryption_key (key)
Set the key used for low-level message encryption. The key is either trimmed or padded automatically to be exactly 32 bytes. If the key value is nil, the encryption is disabled.

Parameters:

Returns:

    nil

Subscription Functions

A subscription is implemented with Tarantool built-in fiber.cond objects.
subscribe ()
Subscribe for updates in the members table.

Returns:

    fiber.cond object which is broadcasted whenever the members table changes
unsubscribe (cond)
Unsubscribe from membership updates. Remove subscription on cond object.

If parameter passed is already unsubscribed o invaled nothing happens.

Parameters:

  • cond fiber.cond object obtained from subscribe function

Returns:

    nil

membership.options Functions

options.PROTOCOL_PERIOD_SECONDS
Period of sending direct PINGs. Denoted as T' in SWIM paper.

Default is 1

options.ACK_TIMEOUT_SECONDS
Time to wait for ACK message after PING. If a member does not reply within this time, the indirect ping algorithm is invoked.

Default is 0.2

options.ANTI_ENTROPY_PERIOD_SECONDS
Period to perform anti-entropy sync. Algorithm is described in SWIM paper.

Default is 10

options.SUSPECT_TIMEOUT_SECONDS
Timeout to mark suspect members as dead.

Default is 3

options.NUM_FAILURE_DETECTION_SUBGROUPS
Number of members to try indirectly pinging a suspect. Denoted as k in SWIM paper.

Default is 3

options.MAX_PACKET_SIZE
Maximum size of UPD packets to send.

Default is 1472 (Default-MTU (1500) - IP-Header (20) - UDP-Header (8))

options.ENCRYPTION_INIT
Initialization vector for aes256 CBC encryption.
generated by LDoc 1.4.6 Last updated 2020-11-18 15:51:38