berkus / kbus Goto Github PK

We're working on splitting up the monolithic source repository into something 
more manageable. This issue exists to track what we're doing / have done.

Motivation: 
(a) It's a bit unfriendly to force users to download all the built HTML 
documentation, talks and slides every time. 
(b) Having the kernel module as its own repo will ease preparation for getting 
it into mainline.
(c) Not everybody wants the C++ and Java interfaces; splitting them out as 
their own module keeps it clear that they're optional.


The dependency tree looks something like this:

kernel module ------ libkbus ---- jkbus
(for kbus_defs.h)
           \-------- cppkbus

The module I call "libkbus" here tentatively contains the init.d, libkbus, 
python and utils subdirs, as well as the docs source.

Separately, the talks and slides will be hived off into their own repo 
(ostensibly named "web") along with the generated HTML docs.

Issues to be resolved are:
* Exactly how far to go with splitting up the repo.
* How to implement source code dependencies. It smells like hg's nested 
repository support could do the job and automate it (i.e.: check out just what 
you want and everything that it needs comes along for the ride) - but such 
things become confusing as soon as you have more than one level of repo (as we 
have, implied from the dependencies above). However there are multiple ways to 
achieve this.

Tasks to perform are:
* Actually split up the repositories, per whatever layout we choose. This is 
straightforward with hg convert, and we probably want to apply the reset option 
to the default repo to clear out the now-transplanted history.
* Update the source buildsystem
* Update HTML buildsystem; rebuild and recommit

Currently Limpets only know about replier bind events that occur subsequent to 
the Limpet being started. This is fine if we assume Limpets being started are 
the first activity on KBus (i.e. before any replier binds), but this is not 
necessarily the case. (The so-called t=0 problem.)

A specific example is a server with many clients repeatedly connecting and 
disconnecting independently, each of which wishes to have a Limpet between it 
and the server. We cannot forbid having any replier binds on the server until 
all the clients are connected (and thus all the Limpets have started) as they 
may never all be connected to the server simultaneously.

One solution would be to have kbus_ksock_report_replier_binds additionally 
inserted a ReplierBindEvent message for every bound replier at the front of the 
calling ksock's message queue, possibly using priority messages. I don't think 
the order in which the messages are inserted would matter.

This solution is neat because it doesn't require any changes to the way Limpets 
work, and if somebody is interested in future replier bind events then they are 
probably also interested in ones that have already happened. The ioctl name 
"KBUS_IOC_REPORTREPLIERBINDS" could probably be left as it is.

What steps will reproduce the problem?
1. make install O=xxx DESTDIR=xxx

* include/kbus/kbus_defns.h is created and include/libkbus/kbus.h is also 
created. shouldn't 
they use one single directory instead of two ?

* "ln -sf /etc/udev/rules.d/45-kbus.rules /lib/udev/rules.d/" doesn't respect 
DESTDIR -- could 
also use a USE_UDEV parameter to skip that since some system don't use/want 
udev.

* libkbus could use a pkg-config .pc file...

* "/kmodules" is not a standard dirctory for installing modules. The ideal 
would be a 
$KERNELDIR/lib/modules/$KERNEL_VERSION/kernel/... in a place that depmod -a 
will find it

The current version of the kernel module uses IOCTLs for all set/query 
purposes.

For "global" values (those which apply to all of the devices open), it 
really should be using a filesystem interface (/proc, /sys. /debugfs or 
whatever seems current best practice) instead.

It is arguable that some of the per-device settings should also be mediate 
in this manner.

Various methods in kbus.py (notably, fcnt.ioctl callers where the result 
is 0 or an exception) return a value when it would be better not to.

Refactor to remove irrelevant returns.

1. The current mechanism for enabling/disabling messages at compile time 
is not perfect - it would be nice to be able to enable/disable at run time.

2. When disabled, there are still too many messages output, with too 
little context to be useful.

3. When enabled, there are many messages which could be removed (as the 
code they were testing is now deemed, well, reliable).

The current Message (Reply, etc.) constructors all allow the first argument to 
be a data-string, as well as a message name (etc).

Consider instead simplifying the constructor to the obvious (first arg is 
message name), and have "from_data" class methods (or some such) for the more 
unusual case.

This should simplify the code, but shouldn't (I think) hurt any use cases.

A sender should be able to send a message to a process that does not yet 
exist.

  (nb: in the following, "KBUS" means "the KBUS kernel module")

The proposed mechanism is to "mark" that this is wanted, indicating what 
process is wanted, and then have KBUS use the existing hotplug (udev) 
mechanisms to start the process (thus taking as much advantage as possible 
of known-working technology).

It may be that in order not to "lose" the initial message (which we're 
assuming the new process will want to receive) KBUS should also bind the 
new KSock for the process and tell it about it. This needs thought.

Question: should this *require* the new process to be started, or do we 
want a mode for "start a process if there is no existing listener"? Maybe 
both.

The build script for libkbus.so should generate a file with the major and minor 
for consistency with ldconfig's expectations.

E.g.  rather than creating libkbus.so, the result shoudl be libkbus.so.0.3, 
with the soname and link names created using symbolic links if at all.

This is a suggestion from Michel.

Imagine we have a sender who is sending "$.redraw" messages, each of which 
gives some parameter defining the redraw.

One can conceive of a listener who wishes only to hear the most recent 
redraw message - any messages they "miss" are not of interest.

Thus, if the sender sends:

* redraw 1
* redraw 2
* redraw 3

and the listener then does "get next message", it only wants to read the 
"redraw 3" message, and not the preceding (presumably now irrelevant) 
messages.

This can probably be emulated by just having a loop that reads until there 
are no more messages (and then "using" the last message), but that's 
clumsy, and probably error prone.

Considerations:

1. What happens if the sender flags the message with ALL_OR_WAIT or 
ALL_OR_FAIL (presumably this listener should not be considered when 
judging whether this message can be sent?)

2. What happens if one of the messages is marked URGENT?

As the summary says. This was left out of the original Python limpet, 
because I've been testing it on the same machine. But I'd like the C and 
Python limpets to interwork.

Write Sphinx documentation for the C library, including examples.

The libkbus script for automating header comment and signature extraction 
needs to cope with both C source files, not just libkbus.c

(this is a place holder to remind me to do this!)

Normally if I edit kbus.c and type 'make' I would expect make to rebuild 
everything that depends on it.

I have fixed this in my scratch tree and will graft it back in due course.

Looking at the differences from revision 339 to 338, using the google code 
interface for same, it is clear that the whitespace has become confused.

For instance, if a process attempts to send messages with undefined names (as 
has happened), KBUS writes the following to the log:

  kbus: (send) message name 'undefined' is not allowed

If the sending process's PID was included in the message, this would make it 
easier to diagnose "who" is at fault.

The various diagnostics output to /proc files are know to "truncate" when 
too much data (more than a page) would need to be output. It would be a 
Good Thing to fix this.

After I've killed my emaild process:

 0: 721 R $.m0.net.sendemail

hangs around in /proc/kbus/bindings and I can't rebind to that symbol
despite having killed all the processes that might bind it. Killing any
process that might be stuck sending a message to it doesn't release it either.

I suspect this is another 'failing to release resources when a process dies
unexpectedly' bug.

The Limpet "do things to a message we got or want to send, so it is 
suitable for such" mechanisms need to be disentangled from the "here is a 
network, transmit over it" mechanisms, so that it is simple to write a new 
network mechanism.

How best to do that?

Various KBUS operations use IOCTLs, and in Python these result in IOError 
exceptions, with a "magic number" inside to indicate the particular 
problem (and a standard Unix interpretation of said number, as a string).

At the C level, this is perfectly sane and traditional, but to a Python 
programmer it is not. In the Python context, it would be more helpful to 
turn these IOError exceptions into something closer to the *actual* 
explanation (as descrined in the KBUS documentation).

Rationale: well, it confuses me, and I wrote the code.

We had a discussion today which almost decided it needed some new data in 
a KBUS message header.

As soon as Limpets are working, we'll be transmitting KBUS messages 
between machines, and thus will really want to have the message header 
frozen (since it will thereafter need a change in guards to indicate a new 
message format).

This thus seems like a sensible time to add a "get out of jail free" space 
to the header.

Question: does KBUS zero it, or does it leave it alone if the user has put 
stuff in it?

When reading a message from KBUS, an "entire" message is always returned
(if it tried to return a "pointy" message, life would be very unsafe).

When writing a message to KBUS, a "pointy" message of any size may be
written, but an "entire" message is limited in size. This is mainly because
it is much easier to cope with a "pointy" message inside the "write" mechanism.

This is excused because "pointy" messages are easier to generate in C, and
because it is always possible to "mock" an "entire" message as "pointy" (by
artificially setting the name and data pointers to the location of the name
and data in the "entire" message).

However, if one is reading data from KBUS, and reflecting it back again
(as, for instance, is done by Limpets), this begins to look silly. The user
should not be required to take extra-ordinary steps (viz, faking an
"entire" message as "pointy") just to make KBUS easier to implement.

All kbuses currently have independent message ids. This means that message 
ids on bus A are meaningless on bus B, so that if (P,R) and Q run on A and 
B, respectively:

P->R: "Foo" msgid=14
P->Q: "Did you get msgid=14"?

Can never work - Q will never know which of its message-ids corresponds to 
msgid=14, so can never figure out if it should wait for msgid 14 or if it 
has missed the message.

This particularly hurts because this situation comes up when waiting for 
daemons to start and in some scenarios involving out-of-band communication 
with multiple busses.

One solution would be to remember a set of (network-id, last-msg) pairs. 
One could then infer that "the current message-id must have happened after 
(network-id,last-msg)" and if Q ever saw, say, (A, N) where N >= 14 in its 
set, it would know that message-id 14 had happened and that if it were 
going to receive it at all, it would've by now.

As a first step towards other things (particularly, the cross-network KBUS 
support), implement a user-space alternative to the KBUS kernel module.

The first implementation should probably be in Python, and should be able 
to use the same (or a subset of the same) unit tests as the Python 
bindings for the kernel module system.

A C version should then be relatively easy to produce, and this may form 
the basis for the networking infrastructure.

When kbus is built into the kernel, i.e. not as a module, there is (at present) 
no way to have it expire unused unloved devices. This leads to some test 
failures, which are expecting to be able to rmmod and have them go away.

We might fix this by having the RESET ioctl clean up and remove all but kbus0? 
I suppose it ought to require some sort of priviliges, e.g. current_uid()==0.

Various Python methods in Message return the actual flags, instead of True/
False. The latter would be more useful.

All of the doctests in the kbus.messages module use the __repr__ of 
messages to illustrate what is going on. This is a good thing, but they 
should also use __str__ for examples, since this actually makes it clearer 
when a message "acts as" an Announcement, Reply, etc., even if it is not a 
specific instance of that particular class.

And if that is deemed to lead to confusion, then it is a confusion that 
needs fixing...

At last nights CamPUG meeting, Bryan Cole suggested the idea of allowing 
multiple repliers for a message name, only one of whom will actually get a 
particular request.

This could allow KBUS to do some sort of load balancing, and sounds like 
an interesting concept which would not (at least in its simple form) be 
hard to implement.

So, several Listeners could bind as Load Balancing Repliers (that needs a 
better name) for a particular message name. When the sender sends a 
Request with that name, KBUS chooses one of the LBRs (sorry) to receive 
the message. The simplest algorithm would be to choose the one with the 
shortest messaage queue, and otherwise one at "random".

A more ambitious approach might allow the choice/specification of policy 
for this choice, by some mechanism.

It is not clear from the C library function documentation (in the C 
library include file, lib/kbus.h) that the "create_entire" message 
functions are fraught with danger - specifically, that they let you create 
an entire message of any length, which the KBUS kernel module may then 
reject as too long.

The intent is that messages being sent should be "pointy" messages (which 
have no such limit).

To address this:

1. rename kbus_msg_create_entire as kbus_msg_create_short
2. rename kbus_msg_create_entire_reply as kbus_msg_create_short_reply
3. enforce the length limit in the C library function (it's with respect 
to a #define from the kernel header file, so that's not too bad a code 
duplication)
4. en route, change the (rather arbitrary) limit of 2000 bytes to 
(slightly less arbitrary) 2048
5. document the two "short" functions as (a) having this length 
limitation, and (b) as not being the preferred way to create messages 
(really, it's only sensible if the "copies the strings" part of their 
functionality is needed).

As of kernel 2.6.36, the ioctl() file operation has been removed, and the newer 
unlocked_ioctl() must be used. This is referenced at 
http://lwn.net/Articles/400074/.

The kernel module should be updated to support the appropriate file operation, 
depending on what the kernel it is being built for wishes.

See this LWN article:

http://lwn.net/Articles/119652/

for what needs doing.

The only use of EFAULT in kbus.c that is *not* either ioctl mandated or 
truly fatal is when the Sender for a Reply cannot be found. This has a 
comment indicating that this is probably not a good response - it is not.

Some more reasonable error should be returned, and a unit test should be 
made for this missed case!

The requirement is to provide a way for messages to be "proxied" between 
two distinct KBUS devices, either on the same machine or two different 
machines.

The proposal is to develop a Limpet process, which proxies a KBUS Ksock to 
another Limpet, doing the same on another KBUS.

Communication between the Limpets is by socket.

This also requires adding Replier Bind Events to KBUS - i.e., a way to 
(optionally) generate automatic messages when a Replier binds or unbinds.

At the moment, the number of KBUS devices is determined when the KBUS 
kernel module is installed (by "insmod" or whatever).

It would be good to be able to do this at run time.

It would also be good to be able to use file permissions to restrict user 
access to a KBUS device (i.e., only able to use it if the permissions 
allow).

Consider use of Limpets to allow communication between different KBUS 
devices (on the same machine or different machines).

Then consider trying to do a stateful transaction (see http://
kbus.googlecode.com/svn/trunk/docs/_build/html/docs.html#id31) through 
such a bridge.

Since the user is not expected to be able to tell which apparent 
connections use Limpets to mediate the message passing and which do not, 
we cannot expect stateful transactions to work for non-Limpet systems and 
not for those using Limpets.

Thus stateful transaction must work through Limpets. But when we are 
sending a message through a Limpet, it currently replaces the originators 
KSock id with that of the Limpet forwarding the message. There is no way 
of producing a useful mapping through this narrowing of information.

This needs resolving now, not so much because we need stateful 
transactions now, but more because we do not want to change KBUS message 
datastructures after the next major release.

The following is not giving a helpful error message!

>>> Message('Fred')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "kbus.py", line 779, in __init__
    self.msg = entire_message_from_string(arg)
  File "kbus.py", line 608, in entire_message_from_string
    return _struct_from_string(local_class, data)
  File "kbus.py", line 343, in _struct_from_string
    thing = struct_class()
MemoryError

If KBUS is being used in any substantial manner, it needs to be Very Quiet 
when operating normally, lest it fill up the system log with irrelevancies.

Comments suggest kbus_msg_delete returns a value, when it does not.

/*
 * Delete a message datastructure.
 *
 * Does nothing if `msg_p` is NULL, or `*msg_p` is NULL.
 *
 * Frees the message datastructure, but does not free any name or data that is
 * pointed to.
 *
 * Returns 0 for success, or a negative number (``-errno``) for failure.
 */
extern void kbus_msg_delete(kbus_message_t **msg_p);

Gareth Bailey discovered an issue when sending requests / receiving 
replies. Broadly, KBUS loses track of the fact that the sender is still 
owed a reply. See attached files...

dmesg.txt shows it clearly.

temp.py appears to be a minimal script to reproduce it in a linear manner.

https://patchwork.kernel.org/patch/291752/ made the change.

If we are only using a sema as a big kbus lock, we probably ought to be using a 
mutex instead. (http://lwn.net/Articles/165039/ refers; it looks like Ingo 
Molnar's work which was accepted.) I have a tentative patch to do this in my 
mainline test repo.

When you bind as both listener and replier to a given name, you get requests 
twice. This is kind of unavoidable, but means that it's hard to avoid taking 
action on a request twice.

Tibs sez there should be a flag to ksock_open() to say 'don't send me things 
I need to reply to as well as everything else'.

When I use kmsg, adding the --dump just after the executable name does not 
change the output.  I still get a summarised version of a reply when I use kmsg 
--dump call <msg name> s <msg data as double quoted string>.

This is problematic, because I want to get a full copy of the results of a call 
operation.

When I run the nosetests on a 64-bit Linux, it fails.

This appears, in the first instance, to be use of struct.unpack("L",...)
instead of struct.unpack("=L",...)

Further, the tests in test_kbus.py fail in various unfortunate ways.

The C library needs reviewing.

The one thing we *know* must be inefficient in the kernel module is the 
way it handles strings.

1. str[n]cmp is going to compare more than is needed, if we just want to 
know if two strings are equal or not
2. There are bound to be places where a simple hash of the string would be 
better (see (1))
3. And I really need to stop copying strings...

I have a program called gozer which binds to a couple of sockets. After
it's died nastily of a segfault:

[885372.819145] gozer[22487]: segfault at 8 ip 0804b733 sp bfa52ba0 error 4
in gozer[8048000+5000]

 I still have:


# cat /proc/kbus/bindings 
 0: 67 L $.m0.gozer.0.command.opensession
 0: 67 R $.m0.gozer.0.command.eval
# cat /proc/kbus/stats 
dev  0: next file 82 next msg 38102
        ksock 67 last msg 0:38095 queue 4 of 100
              read byte 0 of 0, wrote byte 0 (max 0), not sending
              outstanding requests 0 (size 16, max 0), unsent replies 1 (max 1)

 I suspect the unsent reply is stopping the ksock being released. However,
the program which was expecting the reply is now also (obviously) dead.

On a fresh install of Ubuntu 9.10 (karmic koala), the UDEV rules installed 
by the kernel module's Makefile ("sudo make rules") are ignored.

This appears to be because 9.10 now wants the UDEV rules to be in /lib/
udev/rules.d, not in /etc/udev/rules.d

An acceptable fix appears to be to have a symbolic link from one to 
another.

The bindings reporting buffer may not be being cleared properly as it
reports pid's where there is no such process.  E.g. below, the pids 1192,
7752 and 7675 all do not exist when 'ps w' is used to inspect all process ids.

Further, the header comment appears multiple times.  This is relevant,
because the header is supposed to be the start of the information.  This
suggests the items before the last header may be garbage.

# cat /proc/kbus/bindings
# <device> is bound to <KSock-ID> in <process-PID> as <Replier|Listener>
for <message-name>
  0:     9700     1192  L  $.m0.player.0.command.%
  0:     9700     1192  L  $.m0.attentiond.0.event.StreamControls
  0:     9700     1192  L  $.m0.attentiond.0.event.StreamClosed
  0:     9700     1192  L  $.m0.gozer.0.event.%
  0:      688     7752  L  $.m0.gozer.0.event.windowopened
  0:      680     7675  L  $.m0.stb.system.query.%
# <device> is bound to <KSock-ID> in <process-PID> as <Replier|Listener>
for <message-name>
# <device> is bound to <KSock-ID> in <process-PID> as <Replier|Listener>
for <message-name>
# <device> is bound to <KSock-ID> in <process-PID> as <Replier|Listener>
for <message-name>
#

By example:

    with KSock(0, 'rw') as sender:
        with KSock(0, 'rw') as listener:
            listener.kernel_module_verbose(True)
            listener.bind('$.Fred')
            id1 = sender.send_msg(Message('$.Fred'))

            # One message for $.Fred
            assert listener.num_messages() == 1

            listener.bind('$.*')
            id2 = sender.send_msg(Message('$.Fred'))

            # One message for $.Fred, one for $.*
            assert listener.num_messages() == 3

            id3 = sender.send_msg(Message('$.Fred'))

            # One message for $.Fred, one for $.*
            assert listener.num_messages() == 5

            listener.unbind('$.Fred')
            # The documentation says: "Any messages in the listener's
            # message queue which match that unbinding will be removed
            # from the queue". But actually the code is just trying to
            # remove any messages that match the message name.
            #
            # XXX BUG
            # With the current code, we keep all of the messages because
            # they still match $.*. But *some* of them were added for
            # $.Fred and some for $.*, so we really shouldn't be keeping
            # all of them...
            print listener.num_messages()

            listener.unbind('$.*')
            # And the following prints 5 as well!
            #
            # XXX BUG
            # The code in kbus_forget_matching_messages() compares
            # against the "unbind" message name by strncmp() - it doesn't
            # check for wildcard matching. So we don't remove any of the
            # messages because none of them match the string '$.*'. Oops.
            print listener.num_messages()

Fixes:

1. It is wrong that the current code doesn't understand
   comparing message names with wildcards.
2. But the correct solution is to make each item in the
   message queue remember the message and its reason for
   inclusion (i.e., a pointer to the relevant binding
   datastructure). Then when we want to remove messages
   because of unbinding, we can remove them iff they "belong"
   to the binding we're removing. Which may make (1) moot, as
   we may not need to compare on message name at all...

Now, consider, what if we bind to $.Fred twice?
Should we:

a. not allow that
b. silently ignore the second binding (ick)
c. allow it, and mark each message with which binding
   it belongs to, so we can remove only the messages
   we need to when we unbind

(a) feels wrong (although it might be simplest!), since
we're already allowing "overlapping" bindings with wildcards.

(b) just sounds plain wrong, since silent ignoring is the worst
of all worlds.

(c) sounds nice, and comes for free with the Fix proposed
above, but what if we:

   1. bind to $.Fred
   2. get lots of $.Fred messages in our queue
   3. bind to $.Fred again
   4. get one more $.Fred message in our queue
   5. unbind from $.Fred

Does that choose the most recent or the earlier unbinding
to "undo"? (in other words, how many messages does it
remove). Hmm - I can argue either case, so it's probably
sensible to choose whichever the implementation suggests (!),
so long as it is predictable...

Message data is reference counted, so that the kernel module does not make 
N copies for N listeners.

Message names should be treated in the same manner. This should reduce the 
amount of memory used (a good thing), and may also make KBUS slightly 
faster (also a good thing).

The intent of this issue is *not* to worry about making all occurrences of 
a particular string "the same", it is limited to the name of a each 
message individually (doing full "optimisation" of message names might be 
a good thing, but it's a lot more work).

What will happen when a reply is made to a request with a different message 
name to that of the original request is currently undefined.