An overview of Tangence
=======================
Tangence is all of the following:
1. A single server/multiple client protocol for sharing information
about objects.
2. A data model - it defines the types of values that are transmitted
between the server and clients.
3. An object model - it defines the abstract look-and-feel of objects
that are visible in the server end, and the proxies to them that
exist in the client ends.
4. A wire protocol - it defines the bits down the wire of some stream.
5. A collection of Perl modules (a Perl distribution) which implements
all of the above.
These writings may sometimes suffer the "Java problem"; the problem of
the same name being applied to too many different concepts. I'll try to
make the context or wording clear to minimise confusions.
1. Server/Client
----------------
In a Tangence system, one program is distinct in being the server. It is
the program that hosts the actual objects being considered. It is also
the program that holds the networking socket to which the clients
connect.
The other programs are all clients, which connect to the server. While
each client is notionally distinct, they all share access to the same
objects within the server. The clients are not directly aware of each
other's existence, though each one's effects on the system may be
visible to the others as a result of calling methods or altering
properties on the objects. Internally, the clients will use proxy objects
through which to access the objects in the server. There will be a
one-to-one correspondance between server objects and client proxies. Not
every server object needs to have a corresponding proxy in every client -
proxies are created lazily when they are required.
2. Data Model
-------------
Whenever a value is sent across the connection between the server and a
client, that value has a fixed type. The underlying streaming layer
recognises the following fundamental types of values. Each type has a
string to identify call it, called the signature. These are used by
introspection data; see later.
* Booleans
Uses the type signature "bool".
* Integers, both signed and unsigned, in 8, 16, 32 and 64bit lengths
An integer of unspecified size uses the type signature "int".
Specific sized integers use the type signatures
"s8", "s16", "s32", "s64", "u8", "u16", "u32", "u64"
* Floating-point numbers, in 16, 32 and 64bit lengths
A float of unspecified size uses the type signature "float".
Specific sized floats use the type signatures
"float16", "float32", "float64"
Note that the Intel-specific 80bit "extended double" format is not
supported
* Unicode strings
Uses the type signature "str".
* References to Tangence objects
Uses the type signature "obj".
* Lists of values
Uses the type signature "list(T)" where T is the type signature of its
element type.
* Dictionaries of (string) named keys to values
Uses the type signature "dict(T)" where T is the type signature of its
element type.
* Structured records of values
Uses a type signature giving the name of the structure type.
* For type signatures, there is also the type of "any", which allows any
type.
As Tangence is primarily an interprocess-communication layer, its main
focus is that of communication. The Data Model applies transiently, to
data as it is in transit between the server and a client. A consequence
here is that it only considers the surface value of the types of data,
rather than any deeper significance. It does not preserve self-referential
data, nor can it cope with cyclic structures. More complex shaped data
should be represented by real Tangence objects.
3. Object Model
---------------
In Tangence, the primary item of interaction is an object. Tangence
objects exist in the server, most likely bearing at least some
relationship to some native objects in the server implementation (though
if and when the occasion ever arises that a C program can host a Tangence
server, obviously this association will be somewhat looser).
In the server, two special objects exist - one is the Root object, the
other is the Repository. These are the only two well-known objects that
the client knows always exist. All the other objects are initially
accessed via these.
The client(s) interact with the server almost entirely by performing
operations on objects. When the client connects to the server, two special
object proxies are constructed in the client, to represent the Root and
Repository objects. These are the base through which all the other
interactions are performed. Other object proxies may only be obtained by
the return values of methods on existing objects, arguments passed in
events from them, or retrieved as the value of properties on objects.
Each object is an instance of some particular class. The class provides
all of the typing information for that instance. Principly, that class
defines a name, and the collection of methods, events, and properties that
exist on instances of that class. Each class may also name other classes
as parents; recursively merging the interface of all those named.
Tangence concerns itself with the interface of and ways to interact with
the objects in the server, and not with any ways in which the objects
themselves are actually implemented. The class inheritance therefore only
applies to the interface, and does not directly relate to any
implementation behaviour the server might implement.
3.1. Methods
Each object class may define named methods that clients can invoke on
objects in the server. Each method has:
+ a name
+ argument types
+ a return type
The arguments to a method are positional. The return is a single value
(not a list of values, such as Perl could represent).
Methods on objects in the server may be invoked by clients. Once a
method is invoked by a client, the client must wait until it returns
before it can send any other request to the server.
3.2 Events
Each object class may define named events that objects may emit. Each
method has:
+ a name
+ argument types
Like methods, the arguments to an event are positional.
Events do not have return types, as they are simple notifications from the
server to the client, to inform them that some event happened. Clients are
not automatically informed of every event on every object. Instead, the
client must specifically register interest in specific events on specific
objects.
3.3 Properties
Each object class may define named properties that the object has. Each
object in the class will have a value for the property. Each property has:
+ a name
+ a dimension - scalar, queue, array, hash or object set
+ a type
+ a boolean indicating if it is "smashed"
Properties do not have arguments. A client can request the current value
of a property on an object, or set a new value. It can also register an
interest in the property, where the server will inform the client of
changes to the value.
Each property has a dimension; one of scalar, queue, array, hash, or object
set. The behaviour of each type of property is:
3.3.1 Scalar Properties
The property is a single atomic scalar. It is set atomically by the
server, and may be queried.
3.3.2 Queue and Array Properties
The property is a contiguous array of individual elements. Each element is
indexed by a non-negative integer. The property type gives the type of each
element in the array. These properties differ in the types of operations they
can support. Queues do not support splice or move operations, arrays do.
3.3.3 Hash Properties
The property is an association between string and values. Each element is
uniquely indexed by a null-terminated string. The property type gives the
type of each element in the hash. The elements do not have an inherent
ordering and are indexed by unique strings.
3.3.4 Object Set Properties
The property is an unordered collection of Tangence objects.
Scalar properties have a single atomic value. If it changes, the client is
informed of the entire new value, even if its type indicates it to be a
list or dictionary type. For non-scalar properties, the value of each
element in the collection is set individually by the server. Elements can
be changed, added or removed. Changes to individual elements can be sent
to the clients independently of the others.
Certain properties may be deemed by the application to be important enough
for all clients to be aware of all of the time (such as a name or other
key item of information). These properties are called "smashed
properties". When the server first sends a new object to a client, the
object construction message will also contain initial values of these
properties. The client will be automatically informed of any changes to
these properties when they change, as if the client had specifically
requested to be informed. When the object is sent to a new client, it is
said to be "smashed"; the initial values of these automatic properties are
called "smash values".
[There are issues here that need resolving to move Tangence out from
being Perl-specific into a more general-purpose layer - more on this in
a later email].
4. Wire Protocol
----------------
The wire protocol used by Tangence operates over a reliable stream. This
stream may be provided by a TCP socket, UNIX local socket, or even the
STDIN/STDOUT pipe pair of an SSH connection.
The following message descriptions all use the symbolic constant names
from the Tangence::Constants perl module, to be more readable.
4.1. Messages
At its lowest level, the wire protocol consists of a pair of endpoints to
the stream, each sending and receiving messages to its peer. The protocol
at this level is symmetric between the client and the server. It consists
of messages that are either reqests or responses.
An endpoint sends a request, which the peer must then respond to. Each
request has exactly one response. The requests and responses are paired
sequentially in a pipeline fashion.
The two endpoints are distinct from each other, in that there is no
requirement for a peer to respond to an outstanding request it has
received before sending a new request of its own. There is also no
requirement to wait on the response to a request it has sent, before
sending another.
The basic message format is a binary exchange of messages in the following
format:
Code: 1 byte integer
Length: 4 bytes integer, big-endian
Payload: n bytes
The code is a single byte which defines the message type. The collection
of message types is given below. The length is a big-endian 4 byte integer
which gives the size of the message payload, excluding this header. Thus,
the length of the entire message will always be 5 bytes more. The data
payload of the message is encoded in the data serialisation scheme given
below. Each argument to the message is encoded as a single serialisation
item. For message types with a variable number of arguments, the length of
the message itself defines the number of arguments given.
The stream protocol is designed to be used in situations where the CPU
power of each endpoint is high, but the connection in between may have
high latency, or low bandwidth. It is therefore optimised in favour of
roundtrips and byte count overhead, at the expense of processing power
needed to encode or decode it. One consequence here is that no attempt is
made to align multi-byte values.
4.2. Data Serialisation
The data serialisation format applies recursively down a data structure
tree. Each node in structure is either a string, an object reference, or
a list or dictionary of other values. The serialised bytes encode the tree
structure recursively. Other types of entry also exist in the serialised
stream, which carry metadata about the types, such as object classes and
instances.
The encoding of each node in the data structure consists of a type, a
size, and the actual data payload. The type and size of a node are encoded
in its leader byte (or bytes). The top three bits of the first byte
determines the type:
Type Bits Description
DATA_NUMBER 0 0 0 t t t t t numeric
where 'ttttt' gives the number subtype
DATA_STRING 0 0 1 s s s s s string
DATA_LIST 0 1 0 s s s s s list of values
DATA_DICT 0 1 1 s s s s s dictionary of string->value
DATA_OBJECT 1 0 0 s s s s s Tangence object reference
DATA_RECORD 1 0 1 s s s s s structured record
where 'sssss' gives the size
DATA_META 1 1 1 n n n n n
where 'nnnnn' gives the metadata type
For numbers, the lower five bits encode the numeric type, which defines
how many more bytes will be used
Subtype Subtype bits Extra bytes Description
DATANUM_BOOLFALSE 0 0 0 0 0 0 Boolean false
DATANUM_BOOLTRUE 0 0 0 0 1 0 Boolean true
DATANUM_UINT8 0 0 0 1 0 1 Unsigned 8bit
DATANUM_SINT8 0 0 0 1 1 1 Signed 8bit
DATANUM_UINT16 0 0 1 0 0 2 Unsigned 16bit
DATANUM_SINT16 0 0 1 0 1 2 Signed 16bit
DATANUM_UINT32 0 0 1 1 0 4 Unsigned 32bit
DATANUM_SINT32 0 0 1 1 1 4 Signed 32bit
DATANUM_UINT64 0 1 0 0 0 8 Unsigned 64bit
DATANUM_SINT64 0 1 0 0 1 8 Signed 64bit
DATANUM_FLOAT16 1 0 0 0 0 2 Floating 16bit
DATANUM_FLOAT32 1 0 0 0 1 4 Floating 32bit
DATANUM_FLOAT64 1 0 0 1 0 8 Floating 64bit
All multi-byte integers are always stored in big-endian form.
Floating-point values are stored in IEEE 754 form, as three bitfields
containing sign, exponent and mantissa. The sign always has one bit, clear for
positive, set for negative. The exponent and mantissa have the following sizes
and bias.
Subtype Exponent Bias Mantissa
DATANUM_FLOAT16 5 bits +15 10 bits
DATANUM_FLOAT32 8 bits +127 23 bits
DATANUM_FLOAT64 11 bits +1023 52 bits
Infinities and Not-a-Number values are represented by the exponent having its
maximum allowed value. If the mantissa is zero this represents an infinity of
the given sign, and if the mantissa is non-zero, it is a not-a-number value.
For canonical identity, the non-zero mantissa should have only its top bit
set.
Subtype Exponent Mantissa
DATANUM_FLOAT16 31 0 Inf
DATANUM_FLOAT16 31 1 << 9 NaN
DATANUM_FLOAT32 255 0 Inf
DATANUM_FLOAT32 255 1 << 22 NaN
DATANUM_FLOAT64 1023 0 Inf
DATANUM_FLOAT64 1023 1 << 51 NaN
For string, list, dict and object types, the lower five bits give a
number, 0 to 31, which helps encode the size. For items of size 30 or
below, this size is encoded directly. Where the size is 31 or more, the
number 31 is encoded, and the actual size follows this leading byte. For
sizes 31 to 127, the next byte encodes it. For sizes 128 or above, the
next 4 bytes encode it in big-endian format, with the top bit set. Sizes
above 2^31 cannot be encoded.
Following the leader are bytes encoding the data. The exact meaning of the
size depends on the type of the node.
For strings, the size gives the number of bytes in the string. These
bytes then follow the leader.
For lists, the size gives the number of elements in the list. Following
the leader will be this number of data serialisations, one per list
element.
For dictionaries, this size gives the number of key/value pairs. Following
the leader will be this number of key/value pairs. Each pair consists of a
string for the key name, then a data serialisation for the value.
For objects, the size gives the number of bytes in the object's ID number,
followed by a big-endian encoding of the object's ID number. Currently,
this will always be a 4 byte number.
For structured records, the size gives the count of serialied data members for
the record. Following the leader will be the ID number of the structure type
as an int, followed by the given number of data members, in the order that the
structure type declares. The field names are not serialised, as they can be
inferred from the structure type's definition.
Meta-data items may be embedded within a data stream in order to create
the object classes and instances which it contains. These metadata items
do not count towards the overall size of a collection value.
Meta-data operations encode a subtype number, rather than a size, in the
bottom five bits.
Metadata type Bits Description
DATAMETA_CONSTRUCT 1 1 1 0 0 0 0 1 Construct an object
DATAMETA_CLASS 1 1 1 0 0 0 1 0 Create a new object class
DATAMETA_STRUCT 1 1 1 0 0 0 1 1 Create a new record struct type
Following each metadata item is an encoding of its arguments.
DATAMETA_CONSTRUCT:
Object ID: int
Class ID: int
Smash values: 0 or more bytes, encoded per type (in a list container)
If the object class defines smash properties, the construct message will
also contain the values for the smash properties. These will be sent in
a list, one value per property, in the same order as the object class's
schema defines the smash keys. Each will be encoded as per its declared
type.
DATAMETA_CLASS:
Class name: string
Class ID: int
Class: struct of type Tangence.Class
Smash keys: data encoded (list)
The class definition itself will be encoded as a Tangence.Class structure,
containing nested Tangence.Method, Tangence.Event and Tangence.Property
elements. If the class declares any superclasses, these will be sent in
other DATAMETA_CLASS metadata items before this one.
The smash keys will be encoded as a possibly-empty list of strings.
DATAMETA_STRUCT:
Struct name: string
Struct ID: int
Field names: list of strings
Field types: list of strings
4.3. Message Types
Each of the messages defines the layout of its data payload. Some messages
pass a fixed number of items, some have a variable number of items in the
last position. For these messages, no explicit encoding of the size is
given. Instead, the data payload area is packed with as many data
encodings as are required. The receiver should use the size of the
containing message to know when all the items have been unpacked.
The following request types are defined. Any message may be responded to
by MSG_ERROR in case of an error, so this response type is not listed.
Some of these messages are sent from the client to the server (C->S),
others are sent from the server to client (S->C)
MSG_CALL (C->S) (0x01)
INT object ID
STRING method name
data... arguments
Responses: MSG_RESULT
Calls the named method on the given object.
MSG_SUBSCRIBE (C->S) (0x02)
INT object ID
STRING event name
Responses: MSG_SUBSCRIBED
Subscribes the client to be informed of the event on given object.
MSG_UNSUBSCRIBE (C->S) (0x03)
INT object ID
STRING event name
Responses: MSG_OK
Cancels an event subscription.
MSG_EVENT (S->C) (0x04)
INT object ID
STRING event name
data... arguments
Responses: MSG_OK
Informs the client that the event has occured.
MSG_GETPROP (C->S) (0x05)
INT object ID
STRING property name
Responses: MSG_RESULT
Requests the current value of the property
MSG_SETPROP (C->S) (0x06)
INT object ID
STRING property name
data new value
Responses: MSG_OK
Sets the new value of the property
MSG_WATCH (C->S) (0x07)
INT object ID
STRING property name
BOOL want initial?
Responses: MSG_WATCHING
Requests to be informed of changes to the property value. If the
boolean 'want initial' value is true, the client will be sent an
initial MSG_CHANGE message for the current value of the property.
MSG_UNWATCH (C->S) (0x08)
INT object ID
STRING property name
Responses: MSG_OK
Cancels a request to watch a property
MSG_UPDATE (S->C) (0x09)
INT object ID
STRING property name
U8 change type
data... change value
Responses: MSG_OK
Informs the client that the property value has now changed. The
type of change is given by the change type argument, and defines the
data layout in the value arguments. The exact meaning of the operation
depends on the dimension of the property it acts on.
For DIM_SCALAR:
CHANGE_SET:
data new value
Sets the new value of the property.
For DIM_HASH:
CHANGE_SET:
DICT new value
Sets the new value of the property.
CHANGE_ADD:
STRING key
data value
Adds a new element to the hash.
CHANGE_DEL:
STRING key
Deletes an element from the hash.
For DIM_QUEUE:
CHANGE_SET:
LIST new value
Sets the new value of the property.
CHANGE_PUSH:
data... additional values
Appends the additional values to the end of the queue.
CHANGE_SHIFT:
INT number of elements
Removes a number of leading elements from the beginning of the
queue.
For DIM_ARRAY:
CHANGE_SET:
LIST new value
Sets the new value of the property.
CHANGE_PUSH:
data... additional values
Appends the additional values to the end of the array.
CHANGE_SHIFT:
INT number of elements
Removes a number of leading elements from the beginning of the
array.
CHANGE_SPLICE:
INT start
INT count
data... new elements
Replaces the given range of the array with the new elements given.
The new list of values may be a different length to the replaced
section - in this case, subsequent elements will be shifted up or
down accordingly.
CHANGE_MOVE:
INT index
INT delta
Moves the item currently at the index forward a (signed) delta amount,
such that its new index becomes index+delta. The items inbetween the old
and new index will be moved up or down as appropriate.
For DIM_OBJSET:
CHANGE_SET:
LIST objects
Sets the new value for the property. Will be given a list of
Tangence object references.
CHANGE_ADD:
OBJECT new object
Adds the given object to the set
CHANGE_DEL:
STRING object ID
Removes the object of the given ID from the set.
MSG_DESTROY (S->C) (0x0a)
INT object ID
Responses: MSG_OK
Informs the client that the object is due for destruction in
the server. Upon receipt of this message the client should destroy
any remaining references it has to the object. After it has sent the
MSG_OK response, it will not be allowed to invoke any methods,
subscribe to any events, nor interact with any properties on
the object. Any existing event subscriptions or property
watches will have been removed by the server before this message is
sent.
MSG_GETPROPELEM (C->S) (0x0b)
INT object ID
STRING property name
INT|STRING element index or key
Responses: MSG_RESULT
Requests the current value of a single element in a queue or array
(by element index), or hash (by key name). Cannot be applied to
scalar or objset properties.
MSG_WATCH_CUSR (C->S) (0x0c)
INT object ID
STRING property name
INT from
Responses: MSG_WATCHING_CUSR
Similar to MSG_WATCH, requests to be informed of changes to the
property value, which must be a queue property. Creates a new
cursor for the property, beginning at the first index (if
from == 1) or the last (if from == 2).
MSG_CUSR_NEXT (C->S) (0x0d)
INT cursor ID
INT direction
INT count
Responses: MSG_CUSR_RESULT
Requests the next few items from a property cursor. It will yield a
MSG_RESULT message containing up to the given number of items, by
moving forwards (if direction == 1) or backwards (if direction == 2).
If the cursor is already at the edge of the queue then the MSG_RESULT
will contain no extra items.
MSG_CUSR_DESTROY (C->S) (0x0e)
INT cursor ID
Informs the server that the client has finished using the cursor, and it
can release any resources attached to it.
MSG_GETROOT (C->S) (0x40)
data identity
Responses: MSG_RESULT
Initial message to be sent by the client to obtain the root object. The
identity may be used to identify this particular client, as part of its
login procedure. The result will contain a single object reference,
being the root object.
MSG_GETREGISTRY (C->S) (0x41)
[no arguments]
Responses: MSG_RESULT
Requests the registry object from the server. The result will contain a
single object reference, being the registry object.
MSG_INIT (C->S) (0x7f)
INT major version
INT maximal minor version
INT minimal minor version
Responses: MSG_INITED
Requests the start of the Tangence stream. This must be the first message
sent by the client. If the server is unwilling to provide a suitable version
it can return MSG_ERROR. Otherwise, the accepted minor is returned in the
MSG_INITED message.
The version specified by this document is major 0, minor 4.
The following responses may be sent to a request:
MSG_OK (0x80)
[no arguments]
A simple OK message, informing the requester that the operation was
successful, an no error occured.
MSG_ERROR (0x81)
STRING error message
An error occured; the text of the message is included.
MSG_RESULT (0x82)
data... values
Contains the return value from a method call, a property value, or the
initial root or registry object.
MSG_SUBSCRIBED (0x83)
[no arguments]
Informs the client that a MSG_SUBSCRIBE was successful.
MSG_WATCHING (0x84)
[no arguments]
Informs the client that a MSG_WATCH was successful.
MSG_WATCHING_CUSR (0x85)
INT cursor ID
INT first index (inclusive)
INT last index (inclusive)
Informs the client that a MSG_WATCH_CUSR was successful, and returns
the new cursor ID and the first and last indices inclusive of the queue
it will iterate over.
((The reason for using first and last indices inclusively, rather
than yielding the total size of the queue, is that this makes it
easier to support iterating over hashes in a future version))
MSG_CUSR_RESULT (0x86)
INT first item index
data... values
Contains the return value from a MSG_CUSR_NEXT call. Gives the index
of the first item in the returned result, and the requested items.
There may fewer items than requested, if the edge of the property
value was reached.
MSG_INITED (0xff)
INT major version
INT minor version
Informs the client that the initial MSG_INIT was successful, and what
minor version was accepted.
4.4 Built-in Structure Types
The following structure types are built-in, with the given structure ID
numbers. They can be assumed pre-knowledge by both ends of the stream and do
not need serialising by DATAMETA_STRUCT records.
4.4.1 Tangence.Class
Structure ID: 1
Fields:
methods : dict(any)
events : dict(any)
properties : dict(any)
superclasses : list(str)
4.4.2 Tangence.Method
Structure ID: 2
Fields:
arguments : list(str)
returns : str
4.4.3 Tangence.Event
Structure ID: 3
Fields:
arguments : list(str)
4.4.4 Tangence.Property
Structure ID: 4
Fields:
dimension : int
type : str
smashed : bool
5. Perl Distribution
--------------------
The perl distribution is available from
http://bazaar.leonerd.dyndns.org/perl/Tangence/
At some stage when the details become more concrete this will start
gaining inline documentation, but for now it just has some commenting.
As a rough description of the modules:
5.1. Shared by server and client
+ Tangence::Constants
Defines various magic numbers used in the wire streaming protocol.
+ Tangence::Stream
Implements most of the lower level wire streaming protocol, including
the symmetric parts of data serialisation.
5.2. Used by the client
+ Tangence::Connection
The connection to the server. Handles the higher-level client-specific
parts of the wire protocol.
+ Tangence::ObjectProxy
Acts as a proxy to one particular object within the server. Used for
invoking methods, subscribing to events, and interacting with
properties.
5.3. Used by the server
+ Tangence::Object
A base class for implementing Tangence objects within the server.
+ Tangence::Registry
The object registry; keeps a reference to every Tangence object in the
server.
+ Tangence::Server
A base class for implementing the entire server.
+ Tangence::Server::Connection
Server end of a client connection. Handles most of the higher-level
server-specific parts of the wire protocol.
+ Tangence::Server::Context
An object class to represent the client calling context during the
invocation of a server object method or property change.
--
Paul "LeoNerd" Evans
leonerd@leonerd.org.uk
http://www.leonerd.org.uk/