Command-line interactive programs in UNIX shell-scripts

Like it or not, but sooner or later you realize that you’ll have to write shell-scripts to administer UNIX. And among these scripts there certainly will be those to cooperate with interactive applications such as telnet, ftp, su, password, ssh. But it means the end of the admin’s quiet life because while dealing with interactive programs one often come across numerous hidden traps which doesn’t usually happen with ordinary sh-scripts. Though fortunately or may be not, but most of these problems generally
turn up within first five minutes of the work under the script. The symptoms typically look like that author can’t pass the authentication from the script. At first you feel confused because usual pipe constructions such as:

$ echo luser && echo TopSecret | telnet foo.bar.com

fail you and the problem which seemed so plain on the face of
it grows into “mission impossible”. Yet, it isn’t all
so very hopeless and quite a simple solution of the problem will
come quickly enough for many of the dialogue applications include
built-in mechanisms of handling scripts. See, for example,
standard FreeBSD ftp-client:

$ echo '$FILEPUT' | ftp -N ftprc [email protected]

This command will get ftp to connect with the foo.bar.com
hosh with the name of luser and start FILEPUT
macro described in the file ftprc. Besides the macro
there must be also described the host, the login and user’s
password in this file:

$ cat ftprc
machine foo.bar.com
	login luser
	password TopSecret

macdef FILEPUT
	binary
	cd /tmp
	put some_usefull_file.bin
	bye
		<-- Attention! There is a new-line symbol at the end of the macro.
$

If for some reason the dialogue application doesn't support
the built-in scripts then you'll easily find it's freeware
counterpart capable to act automatically. Really you are not the
first to run into such a problem 🙂

Another possibility to persuade an interactive program to do
work by itself, without a user, is to redirect its standard input.
For example, much simplified script to start Oracle database may
look like this:

#!/bin/sh

su - oracle -Ă“ /oracle/bin/svrmgrl <

Yet we can't work like that with all kinds of applications.
For instance, it isn't suitable for telnet, through the

"problem of user's authentication" mentioned above. The
difficulty to debug scripts at the moment of authentication
complicates matters. So, it's clear that to find the way out we
need some special solution. Let's don't break good traditions and
google the Internet. Going through different search systems
brings us some fruits in the form of indistinct mumbling about
the untimely closed I/O data streams, TTYs and PTYs (pseudoterminals)
and all the rest of it. But in all this incongruous abundance
you'll certanly find the links to

expect

It's just what is wanted: the tool, which is traditionally
used to communicate automatically with interactive programs. And
as it always occurs, there is unfortunately a little fault in it:
expect needs the programming language TCL to be present.
Nevertheless if it doesn't discourage you to install and learn
one more, though very powerful language, then you can stop your
search, because expect and TCL with or without TK have
everything and even more for you to write scripts.

If there is no expect installed in your system you
should install it. On FreeBSD you'd better do it by using port-system:

# cd /usr/ports/lang/expect
# make install clean

As a result expect and all it needs will be
downloaded from the Internet and installed on your system.

Now we can work with the TCL and expect. As for the
problem of intercative programs, the expect-script for a short
telnet-session with host foo.bar.com (let it be SCO
UnixWare-7.1.3), under login of luser with password TopSecret

can look like that:

#!/usr/bin/expect

spawn telnet foo.bar.com
expect ogin {send luser\r}
expect assword {send TopSecret\r}
send "who am i\r"
send "exit\r"
expect eof

By the way, the README file of the expect says there
is a libexpect library that can be used to write
programs on C/C++ which allows to avoid the use of TCL itself.
But I'm afraid, this subject is beyond this article. Besides
authors of expect themselves seem to prefer expect-scripts
to the library.

However, if in spite of all the attractiveness of the
foregoing method and all the arguments of the expect
authors (see FAQ) you have made up your mind not to use expect,
then you are either too lazy or entirely poisoned by Perl. Well,
in this case your salvation lies in the installation of the
corresponding Perl-module (http://sourceforge.net/projects/expectperl),
which is supposed to support all the functions of the original expect,
On FreeBSD you can carry it out by the installation from ports:

# cd /usr/ports/lang/p5-Expect
# make install clean

Now our example with telnet-session will be like that:

#!/usr/bin/perl

use Expect;

my $exp = Expect->spawn("telnet foo.bar.com");
$exp->expect($timeout,
        [ 'ogin: $' => sub {
                            $exp->send("luser\n");
                            exp_continue; }
        ],

        [ 'assword:$' => sub {
                                $exp->send("TopSecret\n");
                                exp_continue; }
        ],
        '-re', qr'[#>:] $'
);
$exp->send("who am i\n");
$exp->send("exit\n");
$exp->soft_close();

If I an mistaken and your attachment to Perl isn't very strong
you can take the opportunity of using Python as a corresponding
module pexpect is written for it (http://pexpect.sourceforge.net).
It's clear that Python language should be installed on the system
beforehand, otherwise the FreeBSD ports will help you again:

# cd /usr/ports/lang/python
# make install clean 

And the same for the pexpect module:

# cd /usr/ports/misc/py-pexpect
# make install clean

The script of our telnet-session in Python will be like that:

#!/usr/local/bin/python

import pexpect

child = pexpect.spawn('telnet foo.bar.com');

child.expect('ogin: ');
child.sendline('luser');

child.expect('assword:');
child.sendline('TopSecret');

child.sendline('who am i');
child.sendline('exit');

child.expect(pexpect.EOF);

print child.before;

Certainly, if for some reason Python doesn't suit you either
you can install, let us say, PHP language. Well, I think you
realize that the searching of suitable solution can go on for a
long time and may be only MS Visual Basic will be lacking in the
list of results. So, I believe the time has already approached to
put it all aside and come to

to the Point.

Well, now I'll tell you what really takes place when we start
interactive applications from shell scripts. Though the last
sentence is a bit in the stile of a conclusive speech of Hercules
Poirot we are rather far from the end of narrative. In fact we
are at its beginning. So let's forget everything suggested by expect

and its worthy clones.

At first we'll try to make some rough model to simulate the
expect-like programs. Let's use sh-scripts with all the standard
UNIX tools trying to get our point and the target of this article.
It's essential to note that all the experiments are valid for
FreeBSD and I can't guarantee they'll give the same results on
other operating systems.

To simplify out difficult task and not to fight with pipes
mentioned in the first examples of the article let's make to FIFO-files:
one for the standard input (in.fifo) and the other for the output
(out.fifo) leaving alone the standard error-flow for now:

$ mkfifo in.fifo
$ mkfifo out.fifo

Then with the redirection its I/O to in.fifo and out.fifo let
us execute unsafe telnet of which, I'm afraid, you are already
sick and tired because of numerous examples on expect, perl and
python:

$ telnet -K localhost > out.fifo < in.fifo

A little step aside: as all the experiments are hold on the
FreeBSD box, when we attempt to connect with the FreeBSD telnet
server, SRA secure login mechanisms are automatically involved to
secure telnet-session:

$ telnet localhost
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Trying SRA secure login:
User (luser):			<-- server waits for login, no new-line printed by server

To avoid it and return telnet its traditional
behavior let's start telnet with the -K option:

$ telnet -K localhost
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

FreeBSD/i386 (unity) (ttyp1)

login:				<-- server waits for login, no new-line printed by server

So let our first script (test_1.sh) consist of the following
lines:

#!/bin/sh

mkfifo in.fifo
mkfifo out.fifo

telnet -K localhost > out.fifo < in.fifo 

After executing the script:

$ ./test_1.sh &

we can begin our experiments. Pay attention to the &
parameter which brings the script to the background. It's done
for our possibility to continue working with the same terminal
during the experiments. For example, let's try to read something
from the out.fifo and to write something in the in.fifo:

$ cat out.fifo &

The & parameter plays the same above-mentioned role.
Though, unfortunately the command cat will not show
anything on the screen. Perhaps, it is caused by the blocks on
reading/writing during the work with the FIFO-files: writing may
be blocked till there is no one to read data from FIFO; and
reading is blocked too till the other side isn't ready to write
in it. May be, the whole process is hampered by our in.fifo,
where there is nothing written. Let's check our guess by sending
a newline symbol into the input-channel:

$ echo > in.fifo

Either our guess has been correct or the true reason lies
somewhere else but the miracle has happened! The long-awaited
results of the command cat out.fifo have at last
appeared on the terminal:

$ Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

FreeBSD/i386 (unity) (ttyp1)


login: login:

The only thing that is a little out of the way is the twice-repeated
login:. Actually, that was the natural reaction of the
telnet server on the newline symbol which was sent by echo

> in.fifo command. So let's modify the command echo
> in.fifo
to avoid the additional newline
character:

$ echo -n > in.fifo

May be, it will be even better to use the following
combination:

$ cat > in.fifo &

In the future it will prevent the closing of the input stream,
which is certainly interpreted by telnet server as the connection
breakup.

Well, now we can go on with our research but first we must get
rid of all the background processes, which were caused by using
&. Now run fg command and then send ^C to finish the
process:

$ fg
./test_1.sh
^C[2]  + Done                          cat out.fifo 

The next step will be an attempt to implement the main
functions of expect by filtering the output (out.fifo)
to find the necessary data and sending an answer:

expect request {send answer}

At first glance something like this seems suitable:

$ cat out.fifo | grep request && echo answer

Though in our case this chain won't work correctly even at the
grep point. The matter is the grep is designed
to print strings in accordance with the pattern. So it's quite
natural that before comparing every new line with the pattern grep
always waits for the end either of line ('\n') or file ('\0').
This particular feature makes it impossible to intercept "login:"
with the help of the grep because "login:"

isn't followed by the newline ('\n') character (the system still
waits for user to enter his login-name on the same line). This is
shown above on the example with telnet -K localhost.
Thus grep will be waiting till the end of time for the end
of line
(EOL) and at last on telnet-server timeout it'll see
just the end of file (EOF).

It's clear another way is needed to deal with these unfinished
lines. As a possibility the dd command can be used
instead of cat. In circle the dd will send data
character by character to the grep. I mean the following
construction, which is shown in the example of already working
expect.sh script:

#!/bin/sh

while :; do
        dd if=out.fifo bs=1b count=1 2>/dev/null | grep $1
        if [ $? -eq 0 ]; then
                # Match found
                echo "$2" > in.fifo
                exit 0
        fi
        # Match not found, let's play again
done

The script can be started in this way (files in.fifo and out.fifo
should be already created):

$ ./expect.sh "request" "answer"

So the time has come to gather everything together in one
script to make our traditional telnet-session work automatically:

#!/bin/sh

mkfifo out.fifo in.fifo

telnet -K localhost 1> out.fifo 0< in.fifo &

cat > in.fifo &
cat out.fifo > out.fifo &

pid=`jobid`

./expect.sh "ogin" "luser"
./expect.sh "word" "TopSecret"

sleep 1
echo 'who am i > /tmp/test.txt' > in.fifo
sleep 1
echo "exit" > in.fifo

rm out.fifo in.fifo
kill $pid

After executing this script we may get, in the case of
success, the following output:

$ ./test_2.sh
login:
Password:
Connection closed by foreign host.

And as the war trophy there will appear a file /tmp/test.txt
to confirm the succeeding of our experiment:

$ cat /tmp/test.txt
luser             ttyp3    May 10 16:39 (localhost)

But if something was wrong, the command kill may be
used for each process left after the failed experiment.

Unfortunately, this script is quite unstable: it very much
depends on the value of the sleep parameter and the
reply speed of the telnet-server. So the data don't always come
in time to be properly filtered by dd and grep it
causes script hang-ups. And you must kill these processes. It's
also clear that such a construction doesn't work everywhere, for
example, on Linux I didn't got any acceptable results. So maybe
the adepts of Linux will succeed in it.

As for me

I decided to go on and try to find a tool, which can be used
as an expect-substitute for the pure shell without any
superstructure as TCL, Perl or Python. I realized it must be
written in C and ported for as many operating systems as possible.
Well, let's google! And after some time such a program is found.
It's pty-4.0 written by Daniel J. Bernstein in 1992. But as far
as I can see it haven't been developed after this release.

After a while I even succeeded to compile the source code of
pty-4.0 into binary executable. And some parts of it began to run.
But by that moment I have realized that it's much easier to write
my own program than to sort out the old one.

And why not to write? So I set about studying the problem more
carefully. Before long I found out that the most convenient way
to communicate with interactive applications was really to
imitate a terminal for them. And the place of pseudoterminals in
the structure of the future program was determined clearly
enough, in spite of the contradictive mumbling of specialists
from Internet about expect and PTY-sessions. Next, it
also because clear that it makes everything easy to start
applications under the control of the PTY-sessions inside some
kind of shell, for example TCL, Perl and others. Though nothing
prevent us from using C and pure sh interpreter.

As a result of all this in a couple of weeks I had a working
version of empty (http://www.sourceforge.net/projects/empty)
which allows to start interactive programs and communicate with
them using FIFO-files. For example, the FreeBSD telnet-session in
the sh-script for empty will look like that:

#!/bin/sh

empty -f -i in.fifo -o out.fifo telnet -K localhost
empty -w -i out.fifo -o in.fifo -t 5 "ogin:" "luser"
empty -w -i out.fifo -o in.fifo -t 5 "assword:" "TopSecret"
empty -s -o in.fifo 'who am i > /tmp/test.txt'
empty -s -o in.fifo 'exit'

Well, it's much shorter then the buggy test_2.sh script and
works quite stable on BSD, Linux and Solaris. Besides, it doesn't
require TCL, Perl or Python. I admit there aren't so many
functions in the program yet as there are in other expect-like
tools, but I hope everything is still ahead.


If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.

20 Comments

  1. 2005-06-21 5:59 pm
  2. 2005-06-21 6:51 pm
  3. 2005-06-21 7:09 pm
  4. 2005-06-21 7:12 pm
  5. 2005-06-21 7:23 pm
  6. 2005-06-21 7:25 pm
  7. 2005-06-21 8:12 pm
  8. 2005-06-21 8:46 pm
  9. 2005-06-21 10:32 pm
  10. 2005-06-21 10:42 pm
  11. 2005-06-22 4:26 am
  12. 2005-06-22 6:18 am
  13. 2005-06-22 7:23 am
  14. 2005-06-22 7:42 am
  15. 2005-06-22 11:24 am
  16. 2005-06-22 2:09 pm
  17. 2005-06-22 7:32 pm
  18. 2005-06-22 9:02 pm
  19. 2005-06-29 8:42 pm
  20. 2005-06-30 7:18 am