Net RPC
=======

What found here aims at making RTAI a distributed system, both for kernel and
user space applications. The use of net_ in front of rpc for the name of this
directory main module is due to the existence of an rpc function internally
to all RTAI schedulers, the concept of intertask RPC messaging being strongly 
supported within RTAI since its inception. Clearly nothing new, synchronous 
intertask message passing is now an old concept and the basis of microkernels, 
either distributed or not. So this implementation is nothing but the RTAI 
specific way of doing it. UNIX-Sun RPCs, QNX, PSOS, RTEMS more or less adopted
a similar concept for their distributed usage.

The basic specification for making it easy is:

- use just the name of any already available function 
  substituting rt_... with RT_...

- add two initial arguments more, i.e. the node and the port on the remote 
  machine that will execute rt_... functions that became RT_...

e.g.: 
	
  rt_mbx_send(mbx, msg, msglen);

will become: 

  RT_mbx_send(node, port, mbx, msg, msglen);

Using 0 for the node forces a local execution of the corresponding rt_...
function. In this way you revert to a standard local application
automatically.  So using RT_... naming with a null variable node allows a
local application to be ready for becoming distributed, once you assign a
real node to its variable.  The only added cost will be a longer arguments
list and an if. The node and port arguments will be explained shortly
afterward.

Naturally you can also run a distributed application wholly in local mode by
just setting all your nodes to the local host (127.0.0.1); less effective than 
using a null node but useful for testing the network related stuff.

The only change in args, with respect to the normal local rt_... usage, is
related to any possible argument associated with a time value. It must be
expressed in nanosecs, instead of the RTAI standard internal timer units
count.  In this way you are not asked to know the frequency of any timer
running on a remote machine. These are, more or less, the only things to
know to make any RTAI application distributed. 

There is however a possible usage variation obtained by using -port instead
of port in any call. In fact, as explained further on, a port is a positive
number and making it negative forces an immediate return, i.e an
asynchronous rpc.  It is your responsability to ensure that an asynchronous
rpc is reasonable, e.g. a sem_signal can be used in such a way, while a
sem_wait will likely cause missing a possible synchronization. Notice that
the remote support stub will answer anyhow but the returned values are
discarded. So it should never be used if you want to know the result of
your remote call. Nonetheless, as it will be explained later on, a
mechanism is provided to allow recovering the results of an asynchonous
call left behind.

In any case it is important to know that any further asynchronous call will
not be sent if there is not an answer to the previous one yet. Net_rpc does
not assume any buffering at the remote node, i.e. one can be sure the
remote port is ready to accept calls only if its stub has answered to any
previous service request anyhow. So launching an asynchronous call, if the
previous one has not answered yet, will cause an immediate return without
any action being taken.  In relation to this it is important to note that
the overall maximum  message length allowed, both in sending and receiving
is MAX_MSG_SIZE found in net_rpc.h, actually 1500 bytes, i.e. the number of
octets defined by Linux ETH_DATA_LEN.

The user can check if the port has not answered an asynchronous RPC yet by 
calling:

	void *rt_waiting_return(unsigned long node, int port);

a non null return value implying port at node is waiting an rpc return.
Remote calls that make sense being used asynchronously can be used also in
interrupt handlers, see directory uresumefromintr for an example.

Before doing anything remotely a task working on remote services has to ask 
for a port at its remote peer. Such a port is obtained by a call of the type: 

	myport = rt_request_port(node); 

and released by:

	rt_release(node, myport);

when the port is needed nomore. A task cannot ask for more then one port,
multiple port requests from a task will always return the same port.  The
assigned port will be an integer >= MAX_STUBS, define in net_rpc.h.

Nonetheless a task can create and access more ports by using: 

	anotherport = rt_request_port(node, id);

id being a unique unsigned long id the task must have agreed with any other 
local application. Releasing a port defined with a specific id makes no 
difference with the simpler request above, i.e.:

	rt_release(node, anotherport);

must be used anyhow.

In requesting a port there is also the possibility of providing a mailbox to
recover results of asyncrhounous calls. So you can use either:

	myport = rt_request_port(node, mbx);

or:

	myport = rt_request_port(node, id, mbx);

mbx being a pointer to a mailbox. When a new rpc is made and there is a
result of any previous asynchronous call pending it will be sent to such a
mailbox.  A typical use of this possibility consists in providing a server
task that reads the mailbox and uses the returned values in a manner agreed
with the original RPCs sender. A more direct way to ensure a pending return
is sent to the mailbox is to use:

	int rt_sync_net_rpc(unsigned long node, int port); which forces a
synchronization, thus sending any pending return to a mailbox, if one is
made available at the port request, it returns 1 always.  The above
function allows to recover a pending return immediately, it is likely it
will be used in combination with rt_waiting_return. 

A helper functions is provided to obtain any result queued in the mailbox:

	int rt_get_net_rpc_ret(
		MBX *mbx, 
		unsigned long long *retval, 
		void *msg1, 
		int *msglen1, 
		void *msg2, 
		int *msglen2, RTIME timeout, 
		int type
	);

mbx:		The mailbox
retval:		The value returned by the async call, if any. A double long 
		can contain any value returned by RTAI functions, it is up to
		you to use it properly.
msg1 and msg2:	Buffers for possibly returned messages.
msglen1, 
msglen2:	The length of msg1 and msg2, the helper function return the 
		actual length of the messages, truncating them to msglen1 and 
		msglen2 if their buffers are not larger enough to contain the
		whole returned messages.
timeout:	any timeout value to be used if needed by the mbx_receive 
type:		defined by type, the mailbox receive function to be
		used, i.e.: NET_MBX_RECEIVE for rt_mbx_receive, 
		NET_MBX_RECEIVE_WP for rt_mbx_receive_wp, NET_MBX_RECEIVE_IF 
		for rt_mbx_receive_if, NET_MBX_RECEIVE_UNTIL for 
		rt_mbx_receive_until and NET_MBX_RECEIVE_TIMED
		for rt_mbx_receive_timed.

The function is just a helper, a user can see it as a suggestion for
his/her own optimised implementation, e.g getting just the returned value
or a single message, because he/she knows those are the only returned
values. See the test uasync for a specific example.
 
Even such an asynchronous form of net_rpc does not queue messages, as said
it allows just one effective async call, but can help in increasing the
application parallelism. A full queueing asynchronous form should not be
overly difficult.  It has not been adopted to avoid any assumption on the
possibility of buffering messages at the remote node.

Port requests cause a task rescheduling so they must be called just from
within an RTAI task. Thus care to set up an initializiation task if you
want to use them in a "make it all at once" unified initialisation. Never
call them directly in init_module functions.

Ports request/release need to be timedout, and possibly repeated, in case
of collision on the unique server port used to accept the related requests
on the remote node. A Linux timer is used for this scope, since it is
possible that no RTAI timer is running at the time a request is made. So
ports request/release are assumed to be non real time operations and must
be carried out before beginning any true real time work. Because of this
assumption they are not time critical and can be timed out softly, without
too much a hurry.

A further remark must be made in relation to intertask receive functions.
If you want to receive from any task there is no need to use RT_receive(x),
rt_receive(x) will receive also from remote tasks. If you want a specific
receive instead recall that you will receive from a local stub acting as
the agent of the remote task. RT_receive(x) takes care of it, with a little
added overhead since it has to find the stub associated to the
node-port-task appearing in the RT_receive(x) argument list. You can
improve the efficiency of specific receives from remote tasks by finding
their local stubs yourself, using:

	RT_TASK *rt_find_asgn_stub(unsigned long long owner, int asgn);

where owner combines the remote node and task. It can be built by using the
macro: OWNER(node, task); asgn will assign a port-stub combination to the
owner pair, if it is different from zero and none exists yet. Any following
request for a port from the remote task will be given the one you have
created.  With such a technique you can then effectively use a specific
rt_receive(x) on the task returned by rt_find_asgn_stub. The gain will be
more significant for user space applications, even if it is likely that you
will not notice any difference. 

The same reasoning applies to RT_return(x), take care of using RT_return(x)
with RT_receive(x) and rt_return with rt_receive, never mix them.  The tool
is there, its usage is up to you.

Notice that node must be the networkwise integer corresponding to the
remote IP address. In kernel space a function, ddn2nl, is provided to
transform the standard dotted decimal notation, xxx.xxx.xxx.xxx, into such
an integer.  In user space standard libc functions can be used directly,
see the examples. 

The ddn2l prototype is:

	unsigned long ddn2nl(const char *ddn);

a null value being returned for a bad ddn string. A ddn string is assumed
to be bad if any of its dotted field contains a value greater than 0xFF.

The net_rpc driver module can be made to know its local node by being
insmoded either with the parameter <ThisNode="xxx.xxx.xxx.xxx"> or by a
call to: 

	rt_set_this_node(const char *ddn, unsigned long node);

made by the application module before using any net_rpc service.  In the
above call ddn is the dotted decimal notation string and node the
corresponding node. A call with a NULL ddn means that node contains a valid
value while a non NULL ddn implies that the local node is obtained from
converting the related string, node being discarded.

It is important to remark that at the moment our distributed applications
are awaiting the porting of Dave's (Schleef) rtnet to Linux 2.4.xx, i.e. to
RTAI 24.1.xx, to be truly real time. At the moment rtnet services are just
emulated on the standard Linux networking, thus net_rpc temporarely lacks
true real time performances. 

We expect a new Linux 2.4.xx (rtai-24.1.xx) compatible rtnet release soon.
Nonetheless it is already a useful tool, for both kernel and user space
applications, as it already allows developping coordinated distributed
systems.  At the very least the emulation of rtnet in user space nicely
shows RTAI proper sysreqs at work.

It must be noted also that the specific implementation adopted for our RPC
scheme acts along the ideas already used for LXRT. In fact, networking
apart, the remote execution is carried out by a stub (buddy, proxy, agent)
task.  The possibility of expanding its use to any user specific
application is already in place, once more following the ideas used in
extending LXRT.

At the moment there is just an example in kernel space (ktest) and five in
user space (utest, usound, usoundmsg, uresumefromintr, uasync). Clearly if
they work in user land it works also in the kernel one. That's why there is
only one in kernel space. See the READMEs in the related directories.

It is an advanced beta release that implements most of the functions that
are believed to be of interest for distributed applications.  At the moment
this text is the only documentation available. You can look at net_rpc.h
and the cited examples (u/ktest, usound, usoundmsg, uresumefromintr,
uasync) to better understand what can be done with it. 

For your convenience a list of the functions and macros explained in this
README follows. There is clearly no need to explain RT_... services since
they follow the general rule explained at the very beginning.

- int rt_send_req_rel_port(unsigned long node, int port, unsigned long id, MBX *mbx);

- #define rt_request_port(node)           rt_send_req_rel_port(node, 0, 0, 0)
- #define rt_request_port_id(node, id)    rt_send_req_rel_port(node, 0, id, 0)
- #define rt_request_port_mbx(node, mbx)  rt_send_req_rel_port(node, 0, 0, mbx)
- #define rt_request_port_id_mbx(node, id, mbx)  rt_send_req_rel_port(node, 0, id, mbx)

- #define rt_release_port(node, port)  rt_send_req_rel_port(node, port, 0, 0)

- unsigned long ddn2nl(const char *ddn);

- unsigned long rt_set_this_node(const char *ddn, unsigned long node);

#define OWNER(node, task) \
        ((((unsigned long long)(node)) << 32) | (unsigned long)(task))

- RT_TASK *rt_find_asgn_stub(unsigned long long owner, int asgn);

- int rt_rel_stub(unsigned long long owner);

- int rt_waiting_return(unsigned long node, int port);

- int rt_sync_net_rpc(unsigned long node, int port);

- rt_get_net_rpc_ret(
	MBX *mbx, 
	unsigned long long *retval, 
	void *msg1, 
	int *msglen1, 
	void *msg2, 
	int *msglen2, 
	RTIME timeout, 
	int type
  );


As usual ... comments and bugs fixes are welcomed.


What must be installed for distributed user space applications:
---------------------------------------------------------------

insmoded: rtai.o, rtai_shm.o, rtai_sched.o, rtai_lxrt.o, krtnet.o, 
          net_rpc.o ThisNode="xxx.xxx.xxx.xxx"

process executed in background, as a kind of deamon: urtnet &
library: export LD_LIBRARY_PATH=lxrt/lib

It is important to insert krtnet.o first, followed by net_rpc.o then run urtnet 
process in background.

At rmmod do not kill urtnet, it will happen at krtnet rmmod.


What must be installed for distributed kernel space applications:
-----------------------------------------------------------------

insmoded: rtai.o, rtai_shm.o, rtai_sched.o, lxrt/sched_ext/rtai_sched_ext.o,
	  krtnet.o, net_rpc.o ThisNode="xxx.xxx.xxx.xxx"

process executed in background, as a deamon: urtnet &

It is important to insert krtnet.o first, followed by net_rpc.o then run urtnet 
process in background.

At rmmod do not kill urtnet, it will happen at krtnet rmmod.

Naturally you must care of using the complete path names needed to get to their
directories from your application subdir.
