



	   ################################################
	   #                                              #
	   # ##   ## ###### ####### ##    ## ## ##     ## #
	   # ##   ## ##  ## ##      ###   ## ##  ##   ##  #
	   # ##   ## ##     ##      ####  ## ##   ## ##   #
	   # ##   ## ###### ######  ## ## ## ##    ###    #
	   # ##   ##     ## ##      ##  #### ##   ## ##   #
	   # ##   ## ##  ## ##      ##   ### ##  ##   ##  #
	   # ####### ###### ####### ##    ## ## ##     ## #
	   #                                              #
	   ################################################



	 The following paper was originally published in the
	   Proceedings of the Fourth Annual Tcl/Tk Workshop
		   Monterey, California, July 1996




	For more information about USENIX Association contact:

		   1. Phone:	510 528-8649
		   2. FAX:	510 548-5738
		   3. Email:	office@usenix.org
		   4. WWW URL:  http://www.usenix.org







                      Tksh: A Tcl Library for KornShell

                                Jeffrey Korn
                            Princeton University
                              35 Olden Street
                            Princeton, NJ 08544
                            jlk@cs.princeton.edu

Abstract:

This paper describes Tksh, an implementation of the Tcl C library written on
top of the library for the new KornShell (ksh93). Tksh emulates the behavior
of Tcl by using the API that is provided for extending ksh93, which is
similar to the Tcl library in that it allows access to variables, functions
and other state of the interpreter. This implementation requires no
modification to ksh93, and allows Tcl libraries such as Tk to run on top of
ksh93 unchanged, making it possible to use shell scripts in place of Tcl
scripts. ksh93 is well suited for use with Tk because it is backward
compatible with sh, making it both easy to learn and easy to extend existing
scripts to provide a graphical user interface. Tksh is not yet another port
of Tk to another language - it allows Tcl scripts to run without
modification using the ksh93 internals. This makes it possible to combine
Tcl and ksh93, which is useful for writing ksh93 scripts that use components
that have been implemented in Tcl (such as Tk widgets).

Introduction

Tcl[9] is a general-purpose scripting language which can be reused for a
variety of different tools. It is designed to be extensible by allowing
operations to be performed through a C library interface, such as adding new
commands and manipulating variables. Programs such as expect[8] and Tcl-DP
have been developed this way. The most well known and one of the most useful
extensions to Tcl is Tk, which allows graphical user interfaces to be
developed quickly and easily.

One of the reasons that Tcl/Tk is widely used is that developing user
interfaces with high-level scripting languages can often be easier and
faster than using a low-level language such as C or C++. High-level
languages are able to hide many bothersome details such as storage
management from the user.

Tcl/Tk is not the only user interface scripting language available for UNIX
systems. The Desktop KornShell (dtksh, formerly wksh), which is part of the
Common Desktop Environment (CDE), is one alternative. The CDE is being (or
will be) shipped as a standard part of most UNIX systems. dtksh allows users
to develop Motif user interfaces using the new KornShell command and
programming language (ksh93)[2]. Several reasons make KornShell a reasonable
choice to be the underlying scripting language for CDE:

  1. Backward Compatibility - A large number of users have programmed in
     either the Bourne Shell or KornShell. It is possible to write scripts
     without having to learn a new language. Furthermore, existing shell
     scripts can be easily adapted to provide a user interface.

  2. Conformance to Standards - KornShell conforms to the IEEE P1003.2 and
     ISO 9945-2 shell standards[13]. It deals with the issues of
     internationalization such as error messages, collation order, and
     international character classes used in pattern matching.

  3. KornShell is a Powerful Language - ksh93 has a full set of programming
     constructs such as functions, floating-point math, associative arrays,
     loops, and pattern matching. With a rich set of features, having to
     spawn other processes is infrequent, making ksh93's performance good.

  4. Additional Features - ksh93 has additional powerful features that are
     not typically found in other scripting languages. ksh93 is good for
     interactive use, with features such as command line editing and job
     control. It also has features that make it easy to work with processes
     and files, yet is not tied to UNIX and runs on other operating systems.

Despite the strengths that dtksh has as a scripting language, users often
prefer to use Tk rather than the Motif API for developing user interfaces.
It can be harder to develop interfaces using dtksh because the set of
graphics primitives included with dtksh is similar to the interface to Motif
and X Intrinsics, which were designed to be used with a low level language
like C. Since Tk is not designed around these interfaces, it is both easier
to use and portable to other windowing environments (such as Windows and
MacOS). With the growing popularity of Tk, there have been efforts to port
Tk to other programming languages so users of these languages can take
advantage of Tk's features. There are versions of Tk for Perl (TkPerl[1]),
Scheme (STk[4]), Python (Tkinter[3]), and ML (CamlTk[12]). Tksh has come
into existence because of the growing demand for a toolkit with the
strengths of ksh93 as a scripting language coupled with the strengths that
Tk has for building user interfaces.

The approach taken here is similar to the approach taken in STk; Tksh is not
a modification to Tk (as is TkPerl). Instead, it attacks the problem at a
lower level. Since Tk is written using the set of Tcl internals, Tksh is a
layer in between Tk and ksh93 that provides the interface of the Tcl C API
on top of the internals of ksh93.

However, Tksh is significantly different than other Tk ports in several
ways. Tksh allows unmodified Tcl scripts to be evaluated in the KornShell
interpreter, providing compatibility with the large number of existing
Tcl/Tk scripts. Thus, transition from Tcl/Tk to Tksh is easier since scripts
do not need to be rewritten. The execution of a Tcl script shares state with
the ksh93 interpreter, which means that ksh93 and Tcl scripts can be used
together, sharing the same function and name space. For example, a Tcl
script can specify a ksh function to be executed as a callback for a Tk
event. Similarly, a Tksh script can specify a Tcl procedure to be executeded
for a callback. It is therefore possible to use widgets written in Tcl (such
as an open file dialog box) in a Tksh script.

Tksh can also be more convenient than Tcl/Tk for some tasks. For interactive
scripting, features such as command line editing and job control are useful.
Scripts that do a lot of work with processes and files are easy to construct
with Tksh.

The rest of this paper is organized as follows: The next section introduces
the fundamental concepts underlying Tcl and ksh93, emphasizing the
differences and similarities in the languages. Following that, the use and
implementation of Tksh is described. Examples of applications using Tksh
will be given. We conclude with information on future directions of this
work.

Tcl and ksh93

Tcl is generally recognized as an effective scripting language suitable for
writing useful programs. Tcl is often compared to languages such as Perl or
Python, but is rarely compared to programming with shell due to the
misconception that shells are inherently inadequate for writing large
programs. This notion has developed partly because most of the widely used
shells lack important features that are found in languages such as Perl and
Tcl. Another reason for this opinion is that shells are traditionally slow
because so much time is spent in the fork and exec system calls.

ksh93, the most recent version of the KornShell, offers many new features
absent in traditional shells. Functionality is similar to Tcl, and spawning
external programs is infrequently necessary resulting in increased
performance (for example, command substitution for built-in commands does
not create a separate process). Therefore, it is worthwhile to compare ksh93
to Tcl. This section gives an overview of some of the differences and
similarities between the two languages.

C Library Interface

Both Tcl and ksh93 are built on top of a C library that consists of a set of
functions which manipulate the state of the interpreter. This library is
useful for embedding and extending the language. The two libraries provide
roughly the same set of features (such functions to manipulate variables,
add commands, use hash tables and dynamic strings, etc.), but provide
different interfaces.

Interpreters

Tcl has the notion of an interpreter, which is a structure that maintains
the state of a Tcl script such as defined commands and variables, and the
execution stack. Although only one interpreter is used in most Tcl
applications, it is possible to use several interpreters in a single
program. ksh93 does not have the same notion of multiple interpreters, but
does have the capability to create separate name and function spaces.

Commands

In Tcl, commands can be added to an interpreter using the function
Tcl_CreateCommand. Once a command has been created, scripts that call the
command with the given name will result in a call to the given function.
ksh93 allows built-in commands to be added to the interpreter in a similar
manner. In both languages, the function for created commands takes an argv
list along with a pointer for private data. On architectures that support
dynamically linked shared libraries, built-ins can be added at runtime by
using the builtin command in ksh93. Although Tcl 7.4 does not support
dynamic loading of built-ins, Tcl 7.5 has this feature.

Traces

A trace is a function that is associated with an action performed on a
particular Tcl variable. Traces can be set for three types of events: (1)
when the value of the variable is read from, (2) when the value of the
variable is written to, and (3) when the variable is unset (destroyed).
ksh93 has an equivalent notion called disciplines. Discipline functions can
be C functions or shell functions and can be stacked so that multiple
discipline functions apply to a variable. At the shell level, functions
whose names are in the format variable_name.action are discipline functions.
The discipline actions get, set, and unset are permitted for all variables,
and other discipline names can be defined using C code extensions.

Results

Commands in Tcl produce two results. First, there is an integer completion
code that indicates success (with TCL_OK) or failure (with TCL_ERROR).
Second, there is a field in the interpreter structure that points to a
result string. The result string may either be the result of the successful
command or the error message generated by an unsuccessful one. ksh93 is
similar, but uses uses standard output and standard error to return strings.
This model makes it possible for shell functions and built-ins to behave
like external UNIX commands.

Control Structures

Tcl implements control structures such as if, while, for and foreach as
regular commands. They are not handled specially by the language, as Tcl
consists of very few rules for parsing commands. ksh93 has the same set of
control structures, but they are part of the language.

Name Space

Tcl has a single flat name space for each interpreter. Multiple name spaces
can introduced by creating multiple interpreters. ksh93 uses a hierarchical
name space for variables with . as the separator. A compound assignment
statement makes it easy to assign compound variables. For example,

point=(x=3 y=4)

assigns 3 to the variable point.x and 4 to the variable point.y. This method
can be used to define data structures.

Variables

Both Tcl and ksh93 deal primarily with strings, and support associative
arrays. Tcl allows a variable to indirectly reference another variable
through the use of the upvar command. With ksh93, references can be defined
by using a reference variable type. For example, defining nameref foo=bar
will cause each subsequent operation on variable foo to be an operation on
the variable bar. Reference variables make call-by-name reference possible
without requiring the use of eval. ksh93 also has some additional variable
types not present in Tcl. Floating point numbers are supported for
performing arithmetic, so invoking commands such as bc and awk are no longer
necessary. ksh93 also has indexed arrays in addition to associative arrays.

Expressions and Patterns

Both Tcl and ksh93 support C style expressions, including most arithmetic
operators such as << and ^. With Tcl, the command expr is used to evaluate a
string representing a C expression. In ksh93, C expressions may be used
inside ((...)) notation. ksh93 also allows C style assignments such as += in
expressions.

Tcl uses an extended form of regular expressions for pattern matching.
Although shells traditionally provide notation that is far more restrictive
than regular expressions, ksh93 also uses an extended form of pattern
matching that is very similar to Tcl. The pattern matcher in ksh93 includes
a way to negate expressions, which is not currently possible with Tcl (for
example, it is possible to invoke the command ls !(*.o) to list all files
that don't end in .o).

Quoting

The quoting constructs in Tcl are designed to be easily nested. For example,
[...] notation is used for command substitution and {...} is used for
literal quoting. KornShell behaves similarly for command substitution by
introducing the $(...) syntax to replace the `...` syntax from Bourne shell.
ksh93 does not have a way to nest literal strings, but this is generally not
necessary since nested evaluation is much less common in the shell. ksh93
also allows ANSI C character sequences to be expanded with the $'...'
syntax.

Performance

Although it is hard to precisely measure language performance, ksh93 has
been reported to be comparable to the speed of Perl[5]. Testing shows there
is only a 20%CPU-time penalty from using ksh93 as opposed to a totally C
based application[11]. Another study[14] quotes the speed of ksh as being in
between the speed of Tcl and Perl.

Using Tksh

Tksh implements the library interface for Tcl version 7.4, and works with Tk
versions 3.6 and 4.0 (support for Tcl version 7.5 and Tk version 4.1 is in
development). Tksh can be either statically linked with ksh, or used as a
shared library. As a shared library, Tcl functionality can be loaded by
using the ksh93 builtin command to load the initialization command tclinit
which initializes the Tcl library. Tk can be loaded similarly with the
command tkinit.

Once Tk functionality is loaded, all of the the commands from Tk exist as
shell builtins, and ksh93 can accept Tk commands with ksh syntax. For
example, the ``Hello, world'' program can be written as follows:

pack $(message .b -text "Hello, world")

Scripts can be written that take advantage of features in ksh93. For
example, the following command can be used in an interactive shell to create
a button that always displays the current directory. When the button is
clicked, the current directory will be set to the user's home directory:

pack $(button .b -textvar PWD -command cd)

When using Tksh, all Tcl commands are available and can be used by
prepending tcl_ to the corresponding Tcl command. For example, the Tcl
command info has the name tcl_info in ksh93. The exact Tcl names are not
used because many of the commands have conflicting names with ksh93 and UNIX
commands. For example, Tcl contains commands such as while and if, which
conflict with the corresponding keywords in ksh93. Although using Tcl
commands in a ksh93 script is infrequently necessary, doing so can be more
convenient for a programmer familiar with Tcl. The following example uses
the Tcl command trace to create a window that always displays the contents
of the current directory:

pack $(listbox .list -width 20)
tcl_trace var PWD w trace_pwd
function trace_pwd
{
        .list delete 0 end
        .list insert end *
}

A discipline could be used instead of a trace in the above example by
eliminating the tcl_trace command and changing the name of trace_pwd to
PWD.set.

Figure 1 on the next page has the listing of a small Tksh program. The
script displays a window containing the contents of the current directory as
well as a list of visited directories in an interactive shell. Double
clicking on the name of a file will invoke the editor on the selection;
double clicking on a directory will set the current directory to the
selection. If the a command is typed into the shell that causes the current
directory to change, the window is updated. Sample output is shown in Figure
2.

function selectFile
{
        [[ -d "$1" ]] && cd "$1" > /dev/null
        [[ -f "$1" ]] && ${EDITOR-${VISUAL-emacs}} "$1"
}

function PWD.set        # Discipline called when directory changed
{
        nameref dir=.sh.value           # .sh.value is value being stored
        .f.ls.list delete 0 end
        .f.ls.list insert end .. *
        [[ ${VisitedDir["$dir"]} != "" ]] && return
        .f.dirs.list insert end "$dir"
        VisitedDir["$dir"]=1
}

typeset -A VisitedDir   # Associative array
set -o markdirs

pack $(frame .f)
pack $(frame .f.dirname -relief raised -bd 1) -side top -fill x
pack $(frame .f.ls) $(frame .f.dirs) -side left
label .f.dirname.label -text "Current directory: "
label .f.dirname.pwd -textvariable PWD
pack .f.dirname.label .f.dirname.pwd -side left

scrollbar .f.ls.scroll -command ".f.ls.list yview"
listbox .f.ls.list -yscroll ".f.ls.scroll set" -width 20 -setgrid 1
pack $(label .f.ls.label -text "Directory Contents") -side top
pack .f.ls.list .f.ls.scroll -side left -fill y -expand 1

scrollbar .f.dirs.scroll -command ".f.dirs.list yview"
listbox .f.dirs.list -yscroll ".f.dirs.scroll set" -width 20 -setgrid 1
pack $(label .f.dirs.label -text "Visited Directories") -side top
pack .f.dirs.list .f.dirs.scroll -side left -fill y -expand 1
bind .f.dirs.list "<Double-1>" 'cd $(selection get)'
bind .f.ls.list "<Double-1>" 'selectFile $(selection get)'

cd .

                     Figure 1: Listing of Sample Program

[Image]
                  Figure 2: Screen Dump From Sample Program

As the listing of the program indicates, code to set up a user interface
with Tksh tends to look similar to Tcl scripts with different quoting rules.
The parts that differ the most are in the implementation of the script's
functionality.

Evaluation of Tcl Scripts

Tcl code can be embedded into a ksh93 script by using the source built-in
command. By issuing the command source [filename], the file filename (or
standard input if no file is specified) is parsed and interpreted as a Tcl
script. The sourced Tcl script has access to and can change the current
state of the ksh93 interpreter.

Most existing Tcl/Tk scripts can be evaluated using source with no
modification. For example, the widget demonstration that comes with Tk works
without having to make any changes. Because Tksh can interpret Tcl, it is
able to work with the the large number of existing Tcl/Tk scripts. Thus, an
application can be built with Tksh where the main functionality is written
in ksh93, and widgets are used that have already been written in Tcl. The
transition from Tck/Tk to Tksh is therefore a smooth one. For example, if we
were writing a Tksh script that required a dialog box to open file, we could
take code directly from one of the many Tcl implementations rather than
translate it or rewrite it from scratch.

Functions declared with proc will be parsed as Tcl source when they are
subsequently called, even after source returns. For example, if we have the
following script:

function foo
{
        X=37
        print "$(bar test)"
}

source <<'EOT'
proc bar {args} {
        global X
        set X [expr $X + 1]
        return "Proc bar: args: $args, X: $X"
}
EOT

A call to the function foo will result with the following output:

Proc bar: args: test, X: 38

Note that the Tcl procedure bar is able to use and modify the shell variable
X. It is also possible for Tcl source to invoke ksh93 functions and
built-ins.

Mixing Tcl and ksh93

Some Tk commands take arguments that are strings to be interpreted under
certain conditions. For example, the button command allows a string to be
specified (with the -command option) that will be executed when the button
is pressed. A problem arises with this situation because Tksh needs to
decide whether the string should be parsed as a Tcl command or a shell
command. Normally, Tksh will use the current mode of the parser, but this is
not always desirable. Therefore, Tksh has introduced a special notation to
specify the language of the command string.

If a command string has as its first line #!ksh, ksh will be used, and if
the first line is #!tcl, Tcl will be used (otherwise the current parser is
used). Tksh augments the standard bind command in Tk (which binds a command
string to a window event) with a wrapper function that automatically places
the appropriate line to the beginning of the specified string (if one hasn't
already been specified).

Lists

Tcl uses lists to deal with a collection of strings, whereas ksh usually
uses arrays. Thus, Tcl has a built-in set of functions for dealing with
lists and ksh does not. However, ksh does have the facilities that make it
possible to provide lists. A list can be represented as a string if the
elements of the list are quoted in such a way that splitting the list in ksh
(by processing the arguments) yields the elements of the list. For example,
the Tcl list {a {b c} d} corresponds to 'a "b c" d' in ksh. If we were to
invoke the command eval set - 'a "b c" d' , $1 would be a, $2 would be b c,
and $3 would be d.

Tksh provides two different ways to deal with lists, which can be specified
with the built-in command listmode. The first way is to use ksh style lists
when parsing ksh, and Tcl style lists when parsing Tcl (this is the
default). The second way is to always use Tcl style lists. This mode is
useful because some Tcl applications don't use the appropriate Tcl API to
manipulate lists, and assume the syntax of the list instead (for example, a
function would return ``{a {b c}\}'' directly instead of using the C
function Tcl_Merge to create the list from the individual elements.

A Tcl style list can be used in a shell script by using the built-in command
setlist. Depending on the options, the setlist command will convert a Tcl
style list into either a ksh style list, an array, or the positional
parameters ($1, $2, etc.).

Tcl Result Strings

In Tcl, commands return both a string and a completion code. In ksh,
commands return only a completion code. The means by which strings are
returned in ksh is by printing them to standard output. Thus, Tksh prints
the contents of the result string to standard output upon successful
completion of a Tcl command, and prints the result string to standard error
otherwise. This allows command substitution in ksh behave like it does in
Tcl. Since Tcl does not normally print the result of a command, Tksh only
prints the result of a Tcl command if it is called inside command
substitution.

One important difference between command substitution in Tcl versus ksh has
to do with side effects. Command substitution in ksh93 behaves as if the
command was executed in a separate process, which means that anything done
inside substitution cannot change the state of the interpreter (the only
exception to this is that built-in commands can be added). However, this is
not the case with Tcl. Although not generally a problem, this occasionally
leads to some unexpected behavior when substituting a Tcl command within a
ksh command. Future versions of ksh93 are expected to have an option to
allow side effects in command substitution.

Tksh Architecture

Tksh is written in C and consists of around 5,000 lines of source code along
with a few modules that have been taken directly from the source of Tcl.
Most of the code is straightforward and maps the semantics of a Tcl library
function to the corresponding ksh93 call (if such a function exists). In
several cases, there is either no corresponding function in the API, or the
semantics of the similar functions differ in a nontrivial way. This requires
writing code to emulate the correct semantics. Examples of this are
described throughout this section.

The parts of code that have been taken from Tcl without requiring
modification include the hash table and dynamic string libraries, the Tcl
parser, the implementation of most of the builtin commands (such as proc and
set), and a few miscellaneous routines. While ksh93 has a hash table and
string library, it is easier to use Tcl's routines for the moment because
semantics of the libraries differ.

                                   [Image]

                         Figure 3: Tksh Architecture

Variables and Traces

Traces for variables are implemented on top of the discipline mechanism of
ksh93. Although Tcl traces are similar to disciplines, there are a number of
subtle differences in the mechanisms. For instance, when a Tcl trace is
invoked on a variable, all other traces for that variable are turned off for
the duration of the trace. ksh93, on the other hand, allows other
disciplines to execute as long as they are further down on the discipline
stack. Although it would have been possible to implement a separate tracing
facility that was called from the Tcl_GetVar and Tcl_SetVar functions, using
ksh93 disciplines is advantageous for several reasons reasons. If a variable
is read from, written to or unset from a shell script, Tcl_GetVar and
Tcl_SetVar are never called (and thus the traces wouldn't be). However, if
the traces are installed as disciplines, they will be called in this case.
Also, this implementation allows traces and disciplines to be interleaved on
the same variable - a trace can be followed by a discipline which is
followed by another trace.

Evaluation

In order to interpret Tcl scripts in ksh, an internal flag has been added to
Tksh that indicates which language is currently being interpreted. If we are
interpreting ksh code, the Tcl_Eval and Tcl_EvalFile routines effectively
hand off the string to be evaluated to the ksh93 equivalent, sh_eval. If we
are interpreting Tcl code, the Tcl parsing routines are called instead,
which generate an argv list to evaluated. Before the argv list can then be
handed off to sh_eval, command names are mapped appropriately - Tcl command
names such as set and proc are mapped to names that won't conflict with
ksh93 (tcl_set and tcl_proc).

The Tksh built-in command source sets the aforementioned flag to indicate
Tcl code is to be evaluated, and calls Tcl_EvalFile with the argument. When
source has completed, the flag is restored to its original state.

Another modification has been made to the way the Tcl command proc behaves.
Normally, the implementation of the proc command creates a Tcl command with
the given name that corresponds to a C function named InterpProc. When
InterpProc is called, it passes a field of its private data, which contains
the string of the function body, to Tcl_Eval. In Tksh, InterpProc first sets
the flag indicating Tcl code is to be evaluated before calling Tcl_Eval on
the function body. This way, a Tcl function can be called at any time (in
ksh or Tcl code), not just within the source command.

Dynamic Scope

One important difference between ksh93 and Tcl is that Tcl uses dynamic
scoping, while ksh93 uses static scoping (Tcl uses dynamic scoping in order
to access an array defined in an enclosing function; ksh93 uses reference
variables to accomplish this). The Tcl commands upvar and uplevel make use
of such dynamic scoping, and have been rewritten in Tksh. Although the stack
of enclosing scopes is inaccessible at the scripting level of ksh93, they
are accessible through the C language API. Tksh uses these functions to
achieve the correct behavior.

Files and Processes

Tcl has a set of commands that deal with files and processes. This part of
Tcl has been rewritten in Tksh to make use of ksh93's handing of processes
and files. In Tcl, a function called Tcl_CreatePipeline exists to start a
set of processes and communicate with them (this library function is used by
Tcl's exec command). Tksh implements this function by putting together an
equivalent ksh command string from the arguments and evaluating it in the
shell. For example, the Tcl command:

exec cat << "Hello" | tee foo >@stderr

would be translated to the following shell command:

cat <<HUP | tee foo 1>&2
Hello
HUP

In shell, only 10 files can be referred to at a time (by specifying a number
from 0 to 9 in front of a redirection symbol). In Tcl, the number of files
that can be specified at once is bound by the process limit and thus it is
possible to specify more files at once in Tcl scripts. Tksh gets around this
problem by mapping as many Tcl files to descriptors 0 through 9 as possible.
If a file is not mapped, shell descriptors are moved around to map the file
before the command is invoked, and restored afterwards. This implementation
has the advantage that it allows files to be shared by ksh93 scripts and Tcl
scripts, since ksh93's underlying file facilities are used for both.

Events

In Tcl/Tk, execution is driven by an event loop. Tk has a mechanism which is
used to add events which will invoke a given procedure when the event
occurs. ksh93, on the other hand, has it's own event mechanism that is part
of the shell (the newest releases of Tcl/Tk have moved the event loop from
Tk into Tcl).

The library for ksh93 contains a routine called sh_waitnotify which is used
to specify a function to be called when the shell is waiting for either
input or a child process to complete. The notify function should return when
input is ready or when a signal is received (it must also return within
timeout, which is specified). Tksh uses the sh_waitnotify mechanism to run
the Tk event loop. It registers an event for the file descriptor that the
shell is waiting on, as well as an event to occur after the timeout period.
The function Tk_DoOneEvent is repeatedly called until such an event occurs.
Signal handlers are also placed to trigger an event when a relevant signal
occurs.

The event mechanism in ksh93 has a major advantage over Tcl's. If a command
is executed, ksh93 will process events while waiting for the command to
complete. Tcl, on the other hand, will freeze until the child process
terminates. ksh93 is also able to parse its input as it is entered with this
mechanism since it owns the event loop.

Parsing of Tcl Code

The code for Tcl is structured so well that it was possible to place Tcl's
parser directly into Tksh without modification. The Tcl parser goes through
a string and converts it into an argument list, performing any necessary
variable or command substitutions on the way. Variable substitutions are
done by using the library function Tcl_GetVar, which (as mentioned
previously) has been rewritten to use the ksh93 API. Command substitutions
are done by calling Tcl_Eval which recursively calls the parser.

The function Tcl_Eval calls the parser to generate the argument list, and
then looks up the procedure to be invoked in the ksh93's built-in command
table. If the command exists, it is executed with the argument list.

Patterns and Lists

The ksh93 API has a routine which converts a regular expression into a shell
pattern. Shell patterns are more powerful than regular expressions, and the
translation from regular expressions is fairly simple. The Tcl regular
expression functions are implemented by first translating regular
expressions into shell patterns, and then calling the appropriate ksh93
library function.

A string is converted into a ksh93-style list element using the function
sh_fmtq, which returns a shell-quoted version of its input string. A list is
thus created by concatenating quoted versions of each element.

Tcl Commands

Most Tcl commands that deal with internals of the interpreter are
implemented using the API of the Tcl C library. As a result, such commands
work with Tksh without modification.

int Tcl_SetCmd(dummy, interp, argc, argv)
    ClientData dummy;
    register Tcl_Interp *interp;
    int argc;
    char **argv;
{
    if (argc == 2) {
        char *value;
        value = Tcl_GetVar(interp, argv[1], TCL_LEAVE_ERR_MSG);
        if (value == NULL) {
            return TCL_ERROR;
        }
        interp->result = value;
        return TCL_OK;
    } else if (argc == 3) {
        char *result;
        result = Tcl_SetVar(interp, argv[1], argv[2], TCL_LEAVE_ERR_MSG);
        if (result == NULL) {
            return TCL_ERROR;
        }
        interp->result = result;
        return TCL_OK;
    } else {
        Tcl_AppendResult(interp, "wrong # args: should be \"",
                argv[0]," varName ?newValue?\"", (char *) NULL);
        return TCL_ERROR;
    }
}

                       Figure 4: Listing of Tcl_SetCmd

For example, the implementation of the set command is shown in Figure 4. The
command makes use of Tcl_SetVar, Tcl_GetVar, and Tcl_AppendResult. Since
each of these functions are implemented as part of Tksh, there is no need to
make modifications to the set command itself. There are, however, a handful
of Tcl commands that use functions which are not part of the exported
interface. Commands such as info and rename are examples, and have been
reimplemented for use with Tksh.

Current Work

Tksh is currently being used as a foundation to write a graphical debugger
for C called TkCdb. The debugger runs by spawning an execution of a line
oriented debugger such as gdb or dbx and communicating with the process
through a pipe. The debugger uses external programs, such as a graph drawing
program (called dotty[7]) to display data structures. Shell scripts can be
specified to be executed at breakpoints as well.

Tksh is used as the debugging language because of its strengths as an
interactive command language. Debuggers are interactive programs because
commands are usually entered one at a time to a prompt rather than being
placed into a batch script. TkCdb takes advantage of the interactive
facilities of ksh93 such as command line editing and job control. Using the
command line interface to the debugger feels like being in the shell because
it actually is a shell.

Although TkCdb has a rich set of graphical debugging features, it is less
than 1,500 lines of Tksh. The simplicity, which can be attributed to using a
high-level language and a powerful graphical toolkit, makes the debugger
easy to understand, modify and extend. Future plans are to modify the C
debugger to create a Tcl and ksh debugger.

Another project that is under development which uses Tksh is a file browser,
similar to Window's Program Manager and the Finder in MacOS. The file
browser is unique in that it is designed to work with an interactive shell;
commands can be typed into the ksh command prompt that affect the browser,
and actions triggered through the browser interface can affect the shell.
For instance, the user can scroll through a list of the command history and
double click on an entry to repeat that command.

Future Work

Support for Tcl 7.5

A significant number of changes have been made in the newest release of Tcl,
version 7.5. These new features are currently being integrated into Tksh.
One of the features introduced in Tcl 7.5 is a new I/O system, which is
needed to support versions of Tcl on MacOS and Windows. The I/O system will
be implemented for Tksh on top of the I/O system that is part of ksh93,
called sfio[6]. sfio is a portable I/O library that is backward compatible
with the C stdio library. It has functionality similar to the I/O system in
Tcl 7.5, including the ability to expand the system for new file types and
the ability to do nonblocking I/O. The fact that sfio is backward compatible
with stdio is a major advantage.

Tcl 7.5 also provides facilities for loading dynamic libraries. ksh93 has
this as well, so the proper interface should not be difficult to construct.
Another major Tcl 7.5 feature is the ability to use multiple interpreters.
This feature is also being developed in Tksh.

Once the support for Tcl 7.5 is complete, Tksh will be tested on non-UNIX
platforms, such as Windows 95. Since Tksh has no system dependencies (the
only dependencies are on the ksh93 API), it should work on any system that
runs both Tk and ksh93. This includes most UNIX compatible systems, Windows
NT and Windows 95.

Performance

Performance has not been measured yet with any precision. At this stage,
focus is more on functionality than speed. However, when running the set of
included Tk demos on top of Tksh, performance difference is not perceptible.
The same is true when running Tcl scripts. One area in which performance
could make a difference is in the Tcl_GetVar and Tcl_SetVar functions. Other
aspects, such as the parsing of Tcl strings, will not be affected because
the actual Tcl code is being used. Future work will include finding parts of
Tksh in which performance can be improved.

Interface to Tk

Tksh currently provides the same interface to Tk that Tcl does. However, the
use of disciplines in ksh93 makes it possible to provide a more
object-oriented interface as well. For instance, suppose we wish to
associate a function with a button that will be executed when the button is
pressed. Currently, this might be implemented as follows:

button .b -text "Click here" -command foo

function foo
{
    ...
}

Using disciplines, we could do something like the following:

button .b
.b[text]="Click here"
function .b.mouseup
{
    ...
}

Summary

KornShell has many features that make it a suitable choice for writing
scripts. It has functionality similar to both Tcl and Perl, good
performance, and syntax that is less complex than Perl yet requires fewer
levels of nested evaluation than Tcl does. It is also compatible with the
Bourne Shell and conforms to POSIX and international standards. The creation
of an interface to the internals of ksh93 which is equivalent to the
interface to Tcl internals allows ksh93 to be used with existing Tcl
applications. The most notable of these applications is Tk. In addition,
since Tksh can interpret Tcl scripts, existing Tcl/Tk scripts can be used
with Tksh.

Availability

Tcl and Tk are available via anonymous ftp at ftp.smli.com in the directory
/pub/tcl. Source to ksh93 is freely available for academic purposes (from
the author), and binaries are free for non-commercial use. Binaries can be
obtained through the World Wide Web at
http://www.research.att.com/orgs/ssr/book/reuse. For information on
obtaining a copy of Tksh, see the web page
http://www.cs.princeton.edu/~jlk/tksh.

References

1     M. Beattie, ``TkPerl - A port of the Tk toolkit to Perl5'', Very High
     Level Languages Proceedings, USENIX, 1994.

2     M. Bolsky, D. Korn, The New KornShell Command and Programming
     Language, Prentice Hall, 1995.

3     M. Conway, A Tkinter Life Preserver,
     ftp://ftp.python.org/pub/python/doc/tkinter-doc.tar.gz, 1994.

4     E. Gallesio, ``Embedding a Scheme Interpreter in the Tk Toolkit'',
     Proceedings of the Tcl/Tk 1993 Workshop, USENIX, 1993.

5     D. Korn, ``ksh: An Extensible High Level Language'', Very High Level
     Languages Proceedings, USENIX, 1994.

6     D. Korn, P. Vo, ``Sfio: Safe/Fast String/File IO'', Proceedings of
     Summer USENIX Conference, USENIX, 1991.

7     B. Krishnamurthy, Practical Reliable UNIX Software, Wiley, 1995.

8     D. Libes, Exploring Expect, O'Reilly, 1994.

9     J. Ousterhout, Tcl and the Tk Toolkit, Addison-Wesley, 1994.

10    J. S. Pendergrast, Desktop KornShell Graphical Programming,
     Addison-Wesley, 1995.

11    J. S. Pendergrast, Answers to FAQ Questions,
     http://landru.unx.com/~pend/dtksh.html, 1996.

12    F. Pessaux, F. Rouaix, The CamlTk Interface,
     ftp://ftp.inria.fr/lang/caml-light/camltk.dvi.tar.gz, 1995.

13    POSIX - Part 2: Shell and Utilities, IEEE Standard 1003.2, ISO/IEC
     9945-2, IEEE, 1993.

14    S. Raney, ``A comparison of Tcl/Tk, the Desktop KornShell and
     MetaCard'', http:// www.metacard.com, 1996.

15    B. Welch, Practical Programming in Tcl and Tk, Prentice Hall, 1995.

