Tmac(tcl) 1.0 "Tcl macro processor"

NAME

Tmac - flexible and portable macro preprocessor for Tcl source code

SYNOPSIS

package require tmac
namespace import tmac::*
MAC-BLOCK name ?options? ?namedparameter ...? bodyblock
MAC-FILTER name ?options? script ?fixedparameter ...?
MAC-FILTER name ?options? -proc procname procargs procbody
<:name ?parameters?:>
tmeval string
tmexpand string
tmfindexpand string
tmfileio inputfilename outputfilename
tmmac-block macname ?options? ?parameter names? body
tmmac-filter macname ?options? filtercommand ?fixedparameters?
tmproc ?options? procname paramlist procbody
tmsetcomments c1 c2
tmsetdelims d1 d2
tmsource filename
config(comments)
config(killoctal)
config(notfound)

DESCRIPTION

This page describes how to define and use tmac macros in your tcl source code.

Quick Reference

package require tmac
namespace import tmac::tm*


MAC-BLOCK name ?options? ?namedparameter ...? bodyblock
Define macro to emit bodyblock after parameter substitution. MAC-BLOCK is not a Tcl command. It is a string pattern recognized by Tmac.

MAC-FILTER name ?options? script ?fixedparameter ...?
Define macro to emit the output of script given both fixed and invocation parameters MAC-FILTER is not a Tcl command. It is a string pattern recognized by Tmac. In this form, script is a single argument that is eval'd so it must begin with a command word. Fixed parameters, if any are appended before the eval and preceding any number of additional parameters that may be supplied at each invocation of the macro. Legal ?options? are -parse, -lifetime, and -redefine.

MAC-FILTER name ?options? -proc procname argslist procbody
This is an optional form for MAC-FILTER that lets the implmenting proc be specified right along with the macro definition. In this case, no fixed parameters are allowed and the -proc option and its arguments must end the macro definition. Legal ?options? are -parse, -lifetime, and -redefine.

<:name ?parameters?:>
Invoke either a block or a filter macro by name. Note: the surrounding delimiter characters are configurable.

tmeval string
Find defintions, perform expansions and return result of subsequent eval.

tmexpand string
Perform configured expansions and return the resulting string

tmfindexpand string
Find and record macro definitions, then perform configured expansions and return the resulting string

tmfileio inputfilename outputfilename
Macro process contents of the in file and write result to out file. Both definitions and invocations are processed.

tmmac-block macname ?options? ?parameter names? body
Define a block macro by direct call to the tmac package (not by preprocessing).

tmmac-filter macname filtercommand ?parameters?
Define a filter macro by direct call to the tmac package (not by preprocssing).

tmproc ?options? procname paramlist procbody
Define an ordinary Tcl procedure but do macro processing on the body first.

tmsetcomments c1 c2
Set the strings to enclose macro-based comments.

tmsetdelims d1 d2
Set the strings to enclose macro invocations.

tmsource filename
Read filename and process macro definitions and invocations. Then eval the result.

Introduction

Tmac is a preprocessor for Tcl scripts. It’s basic operation is to take an input string (usually a script) and produce an output script with changes and substitutions as dictated by the macros defined and by configuration options. The output is then commonly passed to the the Tcl interpreter with the eval command. For general overview of the Tmac package (and potential benefits) please see the Introduction page. Tmac processes a file, proc body, or string in 3 distinct phases:

  1. Looks for macro definitions or directives. These are recorded and consumed so they do not appear in the output.

  2. On a second pass through, It detects instances where macros are invoked ("called") and replaces each call with macro substitution.

  3. Delivers the output string by

Reference

package require tmac
namespace import tmac::*


MAC-BLOCK name ?options? ?namedparameter ...? bodyblock
Define macro name to emit bodyblock after parameter substitution. MAC-BLOCK is not a Tcl command. It is a string pattern recognized by Tmac. Tmac recognizes a macro definition by scanning input text. The Tcl command info complete is used to identify the end of the macro definition which may extend over as many lines as needed. Scanning is done line-by-line and the final line is not subdivided. The final line should terminate the macro definition and should not include any other code (see example below). The macro name may not contain white space. Macro names are case-sensitive. Macros may be defined in the single global scope or in a temporary local scope (see Macro Scope for details) Definitions may also be made by calling tmmac-block (see below).

 
# Example macro "returns" capitalized 1st element of a list
MAC-BLOCK FirstCap list {[string totitle [lindex @list 0]]}

# Could be invoked variously. For example:
set favorites {tea coffee coke water}
set top <:FirstCap $favorites:>
# -or-
set top <:FirstCap {tea coffee coke water}:>

# BAD & BROKEN WAY TO END A MACRO:
MAC-BLOCK whoops p1 p2 {
puts "@p1 and @p2"
} ;# wrong!
# to fix this remove ";# wrong!"

A block macro can have any number of named arguments. During subsequent expansion, any argument name found in the bodyblock and preceeded by '@' is removed and the value provided in macro call is put in its place. There is no mechanism to "escape" the meaning of @, however @xxx, where xxx does not begin with a parameter name, passes through parameter substitution untouched. If the macro invocation parameter list is shorter than the formal list then the un-supplied values will be substituted with "". If the supplied list is longer than the formal list, the extra elements simply disappear. The -parse option controls how parameter values are recognized and substituted (see below).

Expr parameter expansion

If the name of a formal parameter begins with an asterisk (*), then the parameter is marked for expr expansion and the asterisk is not included in the parameter name. During macro expansion, the value supplied for an expr marked parameter is processed as follows:

  1. If the value is surrounded in {} the braces are removed and the remaining part is passed unchanged into the macro body.

  2. Leading zeros are removed from literal integers so that the expr command will not try to interpret them as octal values. This behavior is configurable, see Global Configuration.

  3. If the value is an ordinary integer it is passed unchanged.

  4. If the value is "end-N" where N is an integer than the whole value is passed unchanged.

  5. If he value is "end-X" where X is any non-integer string then the X portion is processed as in step 5 and replaces X

  6. Otherwise the value is wrapped in an expr command: "[expr {value} ]" and additionally, apparent variable references inside value are prefixed with '$' if they lack it. Expr's function names are excluded from receiving dollar signs. Array names may recieve dollar signs. However, apparent array indexes will not receive dollar signs. For example: "a(max)", under expr substitution will be substituted as "[expr {$a(max)}]".

 
# package require tmac
source tmac.tcl

puts [tmac::tmfindexpand {
    # Illustrate behavior of expr parameter processing flag

    # The index parameter "i" is flagged for expr expansion
    MAC-BLOCK si s *i {[string index @s @i]}

    set i1 2
    set i2 3
    set a(i) 2
    set s 0123456789

    puts "2     <:si $s 2:>"
    puts "5+2   <:si $s 5+2:>"
    puts "end   <:si $s end:>"
    puts "end-1 <:si $s end-1:>"
    puts "end-i1 <:si $s end-i1:>"
    puts "i1    <:si $s i1:>"
    puts "i1+2  <:si $s i1+2:>"
    puts "i1+i2 <:si $s i1+i2:>"
    puts "a(i) <:si $s a(i):>"
    puts "functions <:si $s int(round(5/2.0)):>"

    # Test leading whiteness for expr function recognition
    # This test only valid if  -parse simple was specified
    # in the macro definition above
    # puts "? <:si $s " round(5/2.0)":>"

    # Brace delimited params do not get processed
    # In this case would be an error
    # puts "{a($key)} <:si $s  {a($key)} :>"
}]
# PRODUCES:

    # Illustrate behavior of expr parameter processing flag

    # The index parameter "i" is flagged for expr expansion

    set i1 2
    set i2 3
    set a(i) 2
    set s 0123456789

    puts "2     [string index $s 2]"
    puts "5+2   [string index $s [expr {5+2}]]"
    puts "end   [string index $s end]"
    puts "end-1 [string index $s end-1]"
    puts "end-i1 [string index $s end-[expr {$i1}]]"
    puts "i1    [string index $s [expr {$i1}]]"
    puts "i1+2  [string index $s [expr {$i1+2}]]"
    puts "i1+i2 [string index $s [expr {$i1+$i2}]]"
    puts "a(i) [string index $s [expr {$a(i)}]]"
    puts "functions [string index $s [expr {int(round(5/2.0))}]]"

    # Test leading whiteness for expr function recognition
    # This test only valid if  -parse simple was specified
    # in the macro definition above
    # puts "? [string index $s [expr {" round(5/2.0)"}]]"

    # Brace delimited params do not get processed
    # The code below will generate an error if eval'd
    # puts "{a($key)} [string index $s a($key)]"
WHICH WHEN EVAL'd OUTPUTS:
    2     2
    5+2   7
    end   9
    end-1 8
    end-i1 7
    i1    2
    i1+2  4
    i1+i2 5
    a(i) 2
    functions 3

MAC-BLOCK accepts options to control how macro calls are parsed, the lifetime of the definition and policy on redefing the the macro.

-parse tcl|simple|keepwrap
Controls how tmac finds and processes parameters in a macro call. The default behavior is keepwrap.

-parse simple
tmac interprets whitespace, double quotes, and braces to identify the individual parameters to the macro call. Outer quotes and braces are discarded before the values are substitued into the macro block. Nesting of double quotes by means of \ is not supported. Discarding of (outter) delimiters is the only difference between simple and keepwrap parsing.

-parse keepwrap
As with with -parse simple, tmac interprets white space, double quotes, and braces to find parameter values but it does not discard the delimiters. Keepwrap makes it easer to pass things like a literal string intended for list processing. A single level of quoting may suffice, whereas with -parse simple, nested braces would be required. Inside words, double quotes or braces are passed through unchanged. Dollar signs '$', square brackets, and backslashes! receive no special treatment. Keepwrap is the default parse behavior.

-parse tcl
Uses the Tcl list command to process macros. This is certainly the least useful style in most cases. Use with caution as it permits dynamic evaluation of your parameters outside of the normal code context. In other words, -parse tcl could result in tmac expanding a local proc variable before the proc itself was even defined, let alone called. So if you must use this option, be careful to protect against unwanted [ or $ expansions.
-lifetime global|local
Controls the context in which the macro defintion will be available (and thus how long the definition will exist. (see Macro Scope for details)

-lifetime global
The corresponding macro definition will stored in the global scope and be available for expansion in either local or a global context. This is the default behavior in most cases; tmproc calls are the exception.

-lifetime local
The corresponding macro definition will be stored in a local context and will be discarded when the local context is ended. Local context can be defined in several ways. For example, all the macro definitions encountered during a tmproc call would by default cease to exist after the call returned.
-redefine ok|disallow|warn|error
Controls behavior when the macro definition would over-write an existing definition. The default value is error.

-redefine ok
The corresponding macro definition will silently replace any existing definition.

-redefine disallow
The redefinition attempt will be silently ignored.

-redefine warn
A warning will be output to stderr. The redefinition will then be allowed.

-redefine error
A descriptive Tcl error will be thrown. This is the default behavior.
-oneline
-oneline
If present, -oneline causes all instances of literal newline in the macro body to be replaced with a space character before being emitted into the source stream. This allows lengthy commands like widget options to be written for clarity using line breaks but not requiring backslashes to end each line.


MAC-FILTER name ?options? script ?fixedparameter ...?
MAC-FILTER name ?options? -proc procname procargs procbody
(Please see the Quick Reference section for a description of the differences between these two forms)

Define macro name to emit the (return) value of script (or proc) given both fixed and invocation parameters. MAC-FILTER is not a Tcl command. It is a string pattern recognized by Tmac. Filter macros are defined and recognized similarly to block macros. Their content or body is produced differently. When the macro is expanded, the user-provided script/proc is run and the result of that script becomes the result of the macro without further processing. Filter macros can be created to implement any sort of transformation on code or data. For example, a new form of switch statement could be created via a filter macro. The macro script would then "re-write" the input strings in terms of existing Tcl commands.

Filter macros support ?options? of -parse, -redefine, and -lifetime. Option -oneline is not supported for filter macros.

Filter macros can also be created by calling tmac::tmmac-filter.

 
# Example filter macro transforms and prints its input
# IMPORTANT: the proc joinupper *must* be defined at macro expansion time because
# that is when it will be called. So if the macro is in a file that is tmsource'd,
# then the proc would need to be defined in a file processed earlier file.

#  FILE 1
proc f1proc {args} {
    return [string toupper [string map [list " " _] [join $args] ] ]
}
....

# FILE 2
MAC-FILTER f1 f1proc monkey meat

puts "filter <:f1 i am your leader!:>"

...
# Output will be: 
filter MONKEY_MEAT_I_AM_YOUR_LEADER!

Unlike block macros, filter macros do not have a defined parameter list. A filter macro is first recorded as a string containing its "script" and any following words. When the macro is invoked, the actual arguments are parsed according to the parse setting. This list is then appended to the stored defintion and the result is evaled. The result of the eval is inserted into the calling text. Currently an artificial limit of 100 is set on the number of actual parameters that will be parsed.

<:name ?parameters?:>
Invoke either a block or a filter macro by name. Note: the surrounding delimiter characters are configurable. The macro name is defined as the first string of non-white characters following the opening delimiter as implemented by [scan ... %s]. Parameter-less macros are useful in providing a sort of readonly or const value or block of code that can be defined in 1 place and expanded in many.

tmeval string
Perform macro defintions and expansions and return result of subsequent eval. This is a three phase process; see Macro Processing Phases for details. Macros to expand must be either defined in the string or defined in the global scope before calling tmeval. All errors are propagated to the caller. Since tmeval runs a full cycle of macro detection, processing and eval, it's better to avoid making the same tmeval call repeatedly.

tmexpand string
Perform expansions and return the resulting string

tmfindexpand string
Find and register macro definitions, then perform configured expansions and return the resulting string

tmfileio inputfilename outputfilename
Macro process contents of the in file and write result to out file. Both definitions and invocations are processed. All errors are propagated to the caller.

tmmac-block macname ?options? ?parameter names? body
Define a block macro by direct call to the tmac package (not by preprocessing). Behavior will be identical to that of an embedded macro definition. By using tmmac-block, Tcl source files that are not themselves macro processed can still define macros to be invoked in other files.

tmmac-filter macname ?options? filtercommand ?fixedparameters?
Define a filter macro by direct call to the tmac package (not by preprocssing). The options and effects for the macro will be indentical to those for an "embedded" macro definition. By using tmmac-filter, Tcl source files that are not themselves macro processed can still define macros to be invoked in other files.

tmproc ?options? procname paramlist procbody
Define an ordinary Tcl procedure but do macro processing on the body first. By default, macro defines created during tmproc are stored in a local context. When tmproc returns these macro definitions are deleted. Similarly, macro invocations in tmproc will first check the local macro context and will prefer local to global macros of the same name. tmproc can be forced to use only the global namespace by using option -lifetime global.

It is probably a bad idea to combine file-based processing such as tmsource with tmproc calls in the sourced file. Combining in this way confuses the global vs. local issue and may lead to unexpected behavior.

tmsetcomments c1 c2
Set the strings to enclose macro-based comments. The motivation for macro comments is that they can comment unconditionally, regardless of mismatched braces. This unconditional ("big gulp") commenting can only work reliably if the string containing the comments is first processed by the macro package. This means that commands like tmsource and tmfilio guarantee comments will always work as expected. However, commands like tmproc operate from within the Tcl interp, and therefore can not protect against unmatched braces. Comment processing is disabled by default. To enable comment processing:

set tmac::config(comments) 1

Default comment delimiters are <* and *>.

tmsetdelims d1 d2
Set the strings to enclose macro invocations. These strings are visible in the tmac::config array, but they should not be set directly because they generally must be processed to escape regular expression special characters before they can be used. There is no built-in restriction on what delimiters can be. They can be any length greater than zero and need not "match" nor even be the same length.

Delimiters, should they occur ANYWHERE in expanded strings will be detected and processed. If these delimiter strings occur in a non-macro they will likely create an error. (In some cases it's possible to let the strings pass through if the errant strings are "matched" - see "config(notfound)". Neither Tcl comments nor {} braces nor \ (backslash) can hide macro delimiters. It's probably a very bad ideas to use square brackets.

The default macro invocation delimiters are <: and :>. The default delimiters seem quite rare in Tcl code, though they are part of the (deprecated) regexp syntax for matching word end/begin. The recommended syntax on the re_syntax manpage is now constraint escapes \m and \M.

tmsource filename
Read filename and process macro definitions and invocations. Then eval the result. All errors are propagated to the caller. The source command is performed at the level of the caller of tmsource (uplevel 1 ...).

Macro Processing Phases

Upto 3 phases of processing can occur on a macro-enhanced script (string, proc, or file). Phases can be skipped depending what procedures are called. The phases are:

  1. Find definitions where MAC-* patterns are detected and valid definitions are stored internally and (generally) removed from the input text.

  2. Expand macro invocations or "calls" where macro calls embedded in the Tcl code are found and processed with the results inserted back into the string where they occurred.

  3. Eval the resulting string performed by tmac as desired.
It's natural to view macro actions as happening in sequence with the Tcl interp. But that is wrong. In a typical example of processing a file with tmsource: There is more that could be said on this topic. For the moment please look at the various tm procs as giving options as to how/when/if you want the various phases performed. Also a good idea to start simple and get something working. It is possible to double process a string. notably with a combination like tmsource on a file and having the file itself contain tmproc calls instead of ordinary proc commands. This may well work or may not. You have been cautioned.

Macro Scope

Macro scope, or lifetime, is a factor in both defining and calling macros. The local scope is provided so that ad-hoc or convenience macros can be created (say within a procedure or a file) without any concern for collisions with macros from other parts of the program. So, for example, when a procedure body is processed with tmac::tmproc any macro definitions discovered will be distinct from any global macro by the same name. Likewise, macro expansion in the procedure body will prefer a local definition. If a local definition is not found then global definition will be used.

Local scopes do not nest. There is only 1 local scope active at any time. The following events create a fresh and empty local scope:

Scope, or lifetime, features are not at all required to use tmac. They are provided for those who want to be protected against macro collision or perhaps want same name macros in 2 versions. It is probably better to get familiar with overall macro behavior before trying local scopes. Macro behavior (even when correct) is easily confusing!

Nesting

Nesting or recursive macro calls can be surprising. The provided behavior is as follows:

  1. When defining macros, the body of a macro may contain another macro call. The body of a macro may not contain another macro definition.

  2. If the body of a macro contains a call to another macro, that call is expanded after parameters to the enclosing macro are substituted. This means that parameters to enclosing macros can become parameters to contained macro calls. Macros should not call themselves.

  3. Macro expansions may also nest. Nested expansions are processed from the inside out, meaning the most deeply nested call is expanded first and the outermost surrounding call is expanded last. This means that expansions can provide parameters to other expansions.

Macro Parameters

Macro usage superficially resembles procedure usage in that both "formal" and actual parameters are involved. Formal parameters are names that serve as place holders for the various values that will be supplied across all the invocations of the macro. This section reviews and somewhat expands on aspects discussed in the command reference section above. Only block macros have named parameters because only block macros do parameter substitution. When the macro call is expanded, tmac first parses the macro name. It then checks the parse method for that macro. By default the method will be "keepwrap". Consider,

 
package require tmac
puts [tmac::tmfindexpand {
    # Illustrate various parameter parsing choices
    MAC-BLOCK firstN-a s n {[string range @s 0 [expr {@n-1}] ]}
    MAC-BLOCK firstN-b -parse simple s n {[string range @s 0 [expr {@n-1}] ]}
    MAC-BLOCK firstN-c -parse tcl s n {[string range @s 0 [expr {@n-1}] ]}

    set s "ABC-6666-wxyz"

    # the -a Variant uses default: -parse keepwrap
    puts -a=keepwrap(default)
    puts <:firstN-a "abc 5555 qrts" 8:>
    puts <:firstN-a {abc 5555 qrts} 8:>
    puts <:firstN-a "$s" 8:>
    puts <:firstN-a {$s} 8:>
    puts <:firstN-a $s 8:>

    # -b Variant uses -parse simple. This means
    # the word delimiters are discarded and an extra layer of
    # quoting is needed for the literal string; the $s still works
    puts -b-simple
    # puts <:firstN-b "abc 5555 qrts" 8:>   ;# BROKEN!
    # puts <:firstN-b {abc 5555 qrts} 8:>   ;# BROKEN!
      puts <:firstN-b {"abc 5555 qrts"} 8:> ;# FIXED by double wrap
      puts <:firstN-b {$s} 8:>   ;# if no whitespace, quote/brace irrelevant
      puts <:firstN-b $s 8:>

    # -c Variant uses -parse -tcl (not very nice)
    puts -c=tcl
    # puts <:firstN-c "abc 5555 qrts" 8:>     ;# BROKEN!
      puts <:firstN-c {"abc 5555 qrts"} 8:>   ;# FIXED
      puts <:firstN-c \$s 8:>                 ;# MUST protect $
}]

Produces

 
    # Illustrate various parameter parsing choices
    set s "ABC-6666-wxyz"

    # the -a Variant uses default: -parse keepwrap
    puts -a=keepwrap(default)
    puts [string range "abc 5555 qrts" 0 [expr {8-1}] ]
    puts [string range {abc 5555 qrts} 0 [expr {8-1}] ]
    puts [string range "$s" 0 [expr {8-1}] ]
    puts [string range {$s} 0 [expr {8-1}] ]
    puts [string range $s 0 [expr {8-1}] ]

    # -b Variant uses -parse simple. This means
    # the word delimiters are discarded and an extra layer of
    # quoting is needed for the literal string; the $s still works
    puts -b-simple
    # puts [string range abc 5555 qrts 0 [expr {8-1}] ] ;# BROKEN!
    # puts [string range abc 5555 qrts 0 [expr {8-1}] ] ;# BROKEN!
    puts [string range "abc 5555 qrts" 0 [expr {8-1}] ] ;# FIXED by double wrap
    puts [string range $s 0 [expr {8-1}] ]   ;# if no whitespace, quote/brace irrelevant
    puts [string range $s 0 [expr {8-1}] ]

    # -c Variant uses -parse -tcl (not very nice)
    puts -c=tcl
    # puts [string range abc 5555 qrts 0 [expr {8-1}] ] ;# BROKEN!
    puts [string range "abc 5555 qrts" 0 [expr {8-1}] ] ;# FIXED
    puts [string range $s 0 [expr {8-1}] ]                 ;# MUST protect $

Please see the MAC-BLOCK reference for more discussion of -parse options.

Global Configuration

Tmac keeps global settings in the array tmac::config. The array is initialized when tmac.tcl is loaded:

 
    ## Tmac default configuration
    set config(mactable) global
    set config(opvals,-parse) [list Tcl simple keepwrap]
    set config(opvals,-lifetime) [list global local]
    set config(opvals,-redefine) [list error warn disallowed ok]
    set config(opnames) [list -parse -lifetime -redefine]

    set config(redefine) ok

    # By default, remove leading zeros in expr expanded macro params
    set config(killoctal) 1

    set config(notfound) error ;# values: error, eat, passthru passinner

    # The "big gulp" comment delimiter strings are
    # disabled by default
    set config(comments)   0
    set config(comRE) [tmsetcomments <* *>]

    # We want to avoid wrongly putting $ infront of expr functions
    set config(exprfuncs) [list abs cosh log sqrt acos double log10 \
                            srand asin exp pow tan atan floor rand \
                            tanh atan2 fmod round ceil hypot sin \
                            cos int sinh]

    # The default macro invocation strings
    tmsetdelims <: :>

Most entries are managed by macro options settings and should not be set directly. Possible exceptions are:
config(comments)
If set to 0 macro-based comments are not performed, otherwise all text between macro comments is silently deleted. This works reliably when processing occurs before the Tcl interpreter has seen the input. If, alternatively, the comments occur within a tmproc call, then usual Tcl rules about balanced braces {} will still apply. In other words, macro comments can only hide braces if macro processing preceeds Tcl eval/source.

config(killoctal)
If set true, then tmac removes leading zeros when it processes macro parameters that are marked for expr expansion. If false, no processing is done. The default value is 1 (true).

config(notfound)
Controls behavior when attempting to expand a macro for which no definition is found. Values are

COPYRIGHT

Copyright © 2003 Roy Terry, royterry@earthlink.net