Working with pipes, threads, and processes

Processes, pipes, threads, and system functions

The following functions allow you to interact with the operating system:

! runs a command in the operating system

abort stops all spawned processes

destroy kills a process

exec runs a process and reads from or writes to it

fork launches a newLISP child process thread (Unix)

pipe creates a pipe for interprocess communication

process launches a child process, remap standard I/O and standard error

semaphore creates and controls semaphores

share shares memory with other processes and threads

spawn creates a new newLISP child process

sync monitors and synchronizes spawned processes

wait-pid waits for a child process to end

Because these commands interact with your operating system, you should refer to the documentation for platform-specific issues and restrictions.

! runs a system command and shows you the results in the console. The exec function does a similar job, but it waits for the operating system to finish, then returns the standard output as a list of strings, one for each line:

(exec "ls -Rat /usr | grep newlisp")
;->
("newlisp" "newlisp-edit" "newlispdoc" "newlisp" "newlisp.1"
"newlispdoc.1" "/usr/share/newlisp:"
"/usr/share/newlisp/guiserver:"
"/usr/share/newlisp/modules:" "/usr/share/newlisp/util:"
"newlisp.vim" "newlisp" "/usr/share/doc/newlisp:"
"newlisp_index.html" "newlisp_manual.html"
"/usr/share/doc/newlisp/guiserver:")

As usual, you'll have all the fun of quoting and double-quoting to get your commands through the shell.

With exec, your script will wait until the command finishes before your script continues.

(exec (string "du -s " (env "HOME") "/Desktop"))

You'll see the results only when the command finishes.

To interact with another process running alongside newLISP, rather than wait until a process finishes, use process. See Processes.

Multitasking

All the newLISP expressions we've evaluated so far have run serially, one after the other, so one expression has to finish evaluating before newLISP can start on the next one. This is usually OK. But sometimes you want to start an expression and then move on to another while the first is still being evaluated. Or you might want to divide a large job into a number of smaller ones, perhaps taking advantage of any extra processors your computer has. newLISP does this type of multitasking using three functions: spawn for creating new processes to run in parallel, sync for monitoring and finishing them, and abort for stopping them before they've finished.

For the examples that follow, I'll use this short 'pulsing' function:

(define (pulsar ch interval)
 (for (i 1 20)
   (print ch)
   (sleep interval))
 (println "I've finished"))

When you run this normally, you'll see 20 characters printed, one every interval milliseconds. The execution of this function blocks everything else, and you'll have to wait for all 20, or stop execution using Control-C.

To run this function in a parallel process alongside the current one, use the spawn function. Supply a symbol to hold the result of the expression, followed by the expression to be evaluated:

> (spawn 'r1 (pulsar "." 3000))
2882
> .

The number returned by the function is the process ID. You can now carry on using newLISP in the terminal while the pulsar continues to interrupt you. It's very irritating - dots keep appearing in the middle of your typing!

Start a few more:

> (spawn 'r2 (pulsar "-" 5000))
2883
> (spawn 'r3 (pulsar "!" 7000))
2884
> (spawn 'r4 (pulsar "@" 9000))
2885

To see how many processes are active (and haven't yet finished), use the sync function without arguments:

> (sync)
(2885 2884 2883 2882)

If you want to stop all the pulsar processes, use abort:

 (abort)
true

On MacOS X, try this more entertaining version that uses the resident spoken voice:

(define (pulsar w interval)
   (for (i 1 20)
      (! (string " say " w)) 
      (sleep interval))
   (println "I've finished"))
 
(spawn 'r1 (pulsar "spaghetti" 2000))
(spawn 'r2 (pulsar "pizza" 3000))
(spawn 'r3 (pulsar "parmesan" 5000))

sync also lets you keep an eye on currently running processes. Supply a value in milliseconds; newLISP will wait for that time, and then check to see if the spawned processes have finished:

> (spawn 'r1 (pulsar "." 3000))
2888
> .
> (sync 1000)
nil

If the result is nil, then the processes haven't finished. If the result is true, then they've all finished. Now - and only now, after the sync function runs - the value of the return symbol r1 is set to be the value returned by the process. In the case of the pulsar, this will be the string "I've finished".

I've finished
> r1
nil
> (sync 1000)
true
> r1
"I've finished"
>

Notice that the process finished - or rather, it printed its concluding message - but the symbol r1 wasn't set until the sync function was executed and returned true. This is because sync returns true, and the return symbols have values, only if all the spawned processes have finished.

You could execute a loop if you wanted to wait for all the processes to complete:

(until (sync 1000))

which checks every second to see if the processes have completed.

As a bonus, many modern computers have more than one processor, and your scripts might be able to run faster if each processor can devote its energies to one task. newLISP leaves it to the operating system to schedule tasks and processors according to the hardware at its disposal.

forked processes

There are some lower-level functions for manipulating processes. These aren't as convenient or as easy to use as the spawned process technique described in the previous section, but they provide a few additional features that you might find a use for some day.

You can use fork to evaluate an expression in another process. Once the process is launched, it won't return a value to the parent process, so you'll have to think about how to obtain its results. Here's a way of calculating prime numbers in a separate process and saving the output in a file:

(define (isprime? n)
 (if (= 1 (length (factor n)))
   true))
 
(define (find-primes l h)
 (for (x l h)
   (if (isprime? x)
       (push x results -1)))
 results)

(fork (append-file "/Users/me/primes.txt" 
  (string "the result is: " (find-primes 500000 600000))))

Here, the new child process launched by fork knows how to find prime numbers, but, unlike the spawned processes, can't return information to its parent process to report how many it's found.

It is possible for processes to share information. The share function sets up a common area of memory, like a notice board, that all processes can read and write to. There's a simple example of share in a later chapter: see A simple IRC client.)

To control how processes access shared memory, newLISP provides a semaphore function.

Reading and writing to threads

If you want forked threads to communicate with each other, you'll have to do a bit of plumbing first. Use pipe to set up communications channels, then arrange for one thread to listen to another. pipe returns a list containing read and write handles to a new inter-process communications pipe, and you can then use these as channels for the read-line and write-line functions to read and write to.

(define (isprime? n)
 (if (= 1 (length (factor n)))
 true))
 
(define (find-primes l h)
 (for (x l h)
   (if (isprime? x)
       (push x results -1)))
 results)

(define (generate-primes channel)
 (dolist (x (find-primes 100 300))
  (write-line channel (string x))))         ; write a prime

(define (report-results channel)
 (do-until (> (int i) 290)
  (println (setq i (read-line channel)))))  ; get next prime

(define (start)
 (map set '(in out) (pipe))                 ; do some plumbing
 (set 'generator (fork (report-results in)))
 (set 'reporter (fork (generate-primes out)))
 (println "they've started"))
 
(start)

they've started
101
103
107
109
...

(wait-pid generator)
(wait-pid reporter)

Notice how the "they've started" string appears before any primes are printed, even though that println expression occurs after the threads start.

The wait-pid function waits for the thread launched by fork to finish - of course you don't have to do this immediately.

Communicating with other processes

To launch a new operating system process that runs alongside newLISP, use process. As with fork, you first want to set up some suitable plumbing, so that newLISP can both talk and listen to the process, which in the following example is the Unix calculator bc. The write-buffer function writes to the myout pipe, which is read by bc via bcin. bc's output is directed through bcout and read by newLISP using read-line.

(map set '(bcin myout) (pipe))                 ; set up the plumbing
(map set '(myin bcout) (pipe))
(process "/usr/bin/bc" bcin bcout)             ; start the bc process 
(set 'sum "123456789012345 * 123456789012345")
(write-buffer myout (string sum "\n"))
(set 'answer (read-line myin))
(println (string sum " = " answer))

123456789012345 * 123456789012345 = 15241578753238669120562399025

(write-buffer myout "quit\n")                  ; don't forget to quit!