Handling callbacks into Chicken from other threads

During the work on the sdl-mixer egg I have encountered a limitation with Chicken Scheme's FFI and found a (somewhat hacky) way around it. In this blog post I want to share it with you and put it up to discussion.

When a piece of music or a sample has been played by libsdl-mixer, a callback can be defined to receive a notice of the event and to be able to act upon it.

The chicken extension should also provide this mechanism so the user code can easily queue another file or sample.

Since Chicken allocates everything on the stack using CPS conversion and Cheney on the MTA some precautions need the be taken care of when you want non scheme code call scheme procedures.

First of all we need to take care that callbacks occur only at defined points. In Chicken's FFI there is a macro for wrapping calls to foreign functions that might themselves call back into scheme: foreign-safe-lambda.

Second we need to make our scheme procedure visible to the non scheme world with define-external.

From the callbacks documentation:

(define-external foo int 42)
((foreign-lambda* int ()
  "C_return(foo);"))

So that looks easy, right? It is, but it has one caveat: The callback can only occur when we call a procedure with foreign-safe-lambda. In libsdl-mixer we have to deal with a different beast.

The libsdl-mixer code does decoding of sound data in another (posix) thread. When we set a callback through its API the function pointer gets stored away in a safe location and passed to the decoding threads. Once the event for the callback occurs the function gets called. This means that our callback does get called at the wrong point in time from the scheme's perspective. The result is a panic in the runtime system: "callback invoked in non-safe context".

So it may seem that we are out of luck with this mechanism. We cannot use the traditional FFI procedures for this callback case, yet we need to pass the information that our event has happened back into our chicken runtime.

The first idea has been to implement the callback function for libsdl-mixer as a pure C function which then transfers the event back into scheme. But how? We could poll the state of a global variable and react on this, calling then the scheme procedure callback, but polling is a Bad Thing in this case.

Then it occurred to me that for this situation there is one nice system call available, select(1). And Chicken already uses this in its own scheduler to wake up threads waiting for I/O. The interface for this is the thread-wait-for-i/o! procedure from srfi 18. All it needs is a file descriptor (not a port!) to listen on. Where are we going to get this file descriptor?

Here the posix egg comes to our rescue: We will create a pipe, a unidirectional pair of two file descriptors which we can pass around and read from on the scheme side. The C function in return will write to it when the event occurs.

So the cost of having the callback is spawning a thread which will sleep most of the time and opening a file descriptor as communication channel. That seems to be doable. And it was. Here is what one callback handler looks like:

#>
int channel_callback_fd = -1;
static void channel_finished (int channel){
	if (channel_callback_fd >= 0) {
                char chan[3];
        	snprintf(chan, 3, "%d\n", channel);
                write(channel_callback_fd, chan, 2);
	}
}
<#
        (define-external channel_callback_fd int -1)

	(define mix-channel-finished-scheme-cb #f)

        (define (make-waiter-thread handler)
          (let-values (((in out) (create-pipe)))
            (set! channel_callback_fd in)
            (let ((p (open-input-file* out)))
              (let loop ()
                (thread-wait-for-i/o! out #:input)
                (let ((v (read p)))
                  (unless (eof-object? v)
                          (handler v)
                          (loop))
                  (close-input-port p))))))

        (define (set-mix-channel-finished-cb handler)
          (when (eq? handler mix-channel-finished-scheme-cb)
                (foreign-code "close(channel_callback_fd);"))
          (set! mix-channel-finished-scheme-cb handler)
          (foreign-code "Mix_ChannelFinished(channel_finished);")
          (thread-start! (make-thread (cut make-waiter-thread handler))))

As you can see we will define a global in C holding the C part of the pipe and on the scheme side create a thread that will try to read from one end until we get the #!eof symbol. So to uninstall the callback we simply close the file descriptor.

Also thanks to the read procedure we do not worry about having to implement a parser for the response.

Depending on the API of the callback writing to the file descriptor can be a bit tedious, but I think the ease of use on the scheme side makes up for all the trouble.

So there seems to be a nice way to handle such use cases. Depending on the library just one pair of these file descriptors and one thread might suffice if one adds a dispatching "protocol" on either end to multiplex multiple events through one channel.

I hope this little thing might be useful to you too and I am looking forward to hearing your suggestions!