source: Running One Script From Another

Part I · Module 15 · Chapter 15.1 · source: Running One Script From Another

Everything so far has been one file. That stops working at a few hundred lines: you want the procedures somewhere separate from the script that calls them, so you can read one without the other, and so two programs can share the same helpers.

source is how Tcl does that. It reads another file and evaluates it — as if you had typed its contents at that point. That sentence is the whole command, and it is also the source of both problems in this chapter.

1. The smallest working example

Two files. The library:

# lib/util.tcl
proc average {values} {
    if {[llength $values] == 0} {
        error "average requires at least one value"
    }
    set total 0.0
    foreach value $values {
        set total [expr {$total + $value}]
    }
    return [expr {$total / [llength $values]}]
}

And the script that uses it:

# main.tcl
source lib/util.tcl
puts "average = [average {10 20 30}]"

Run it:

$ tclsh main.tcl
average = 20.0

source lib/util.tcl read the file and ran it. Running it defined average, so the next line could call it. Nothing more mysterious than that is happening.

2. What happens when the file is not there

# miss.tcl
source lib/typo.tcl
puts "never reached"

couldn't read file "lib/typo.tcl": no such file or directory
    while executing
"source lib/typo.tcl"
    (file "miss.tcl" line 1)

It is an ordinary Tcl error, so everything from Module 14 applies — it unwinds, it can be caught, and the script stops if nothing catches it. never reached did not print.

That is usually the behaviour you want. A program missing half its procedures should stop immediately, not fail later with invalid command name "average" in a place that has nothing to do with the real problem.

3. source runs in the current scope

Here is the part the registry purpose is pointing at, and it is worth being precise because the behaviour is asymmetric.

A small file that both sets a variable and defines a procedure:

# defs.tcl
set setting "from the sourced file"
proc helper {} { return "helper ran" }

Sourced at the top level, both are available afterwards:

source defs.tcl
puts "  setting is: [expr {[info exists setting] ? $setting : "MISSING"}]"
puts "  helper     : [helper]"

  setting is: from the sourced file
  helper     : helper ran

Now source the same file from inside a procedure:

proc load_them {} {
    source defs.tcl
    puts "  inside the proc, setting is: [expr {[info exists setting] ? $setting : "MISSING"}]"
    puts "  inside the proc, is it local? [info exists setting] / locals: [lsort [info locals]]"
}
load_them
puts "  AFTER the proc returns:"
puts "    setting at global scope : [expr {[info exists ::setting] ? $::setting : "MISSING"}]"
puts "    helper still defined?   : '[info procs helper]'"

  inside the proc, setting is: from the sourced file
  inside the proc, is it local? 1 / locals: setting
  AFTER the proc returns:
    setting at global scope : MISSING
    helper still defined?   : 'helper'

Read those last two lines together, because they disagree on purpose:

• The variable is gone. set setting ... ran inside load_them's frame, so it created a local, which info locals confirms. When the procedure returned, the local was discarded.
• The procedure is still there. proc always creates a command in the global namespace no matter what scope it runs in, so helper survived.

4. When a sourced file fails halfway

A library that raises partway through leaves you in a worse state than one that fails completely:

# lib/broken.tcl
proc ok {} { return 1 }
error "lib/broken.tcl cannot configure itself"
proc never {} {}

Sourced normally, the error propagates out, and the trace names both files:

before
lib/broken.tcl cannot configure itself
    while executing
"error "lib/broken.tcl cannot configure itself""
    (file "lib/broken.tcl" line 2)
    invoked from within
"source lib/broken.tcl"
    (file "usebroken.tcl" line 2)

That trace is doing real work: the inner frame is the line inside the library, the outer frame is the source call site. When a sourced file breaks, those two lines tell you which file and which caller.

Now catch it and look at what actually got defined:

catch {source lib/broken.tcl} e
puts "  caught: $e"
puts "  is 'ok' defined?    '[info procs ok]'"
puts "  is 'never' defined? '[info procs never]'"

  caught: lib/broken.tcl cannot configure itself
  is 'ok' defined?    'ok'
  is 'never' defined? ''

5. The relative-path trap

This is the single most common way a working Tcl program breaks on somebody else's machine.

main.tcl from §1 works:

$ tclsh main.tcl
average = 20.0

Run the identical script from the directory above, and it does not:

$ tclsh d151/main.tcl
couldn't read file "lib/util.tcl": no such file or directory
    while executing
"source lib/util.tcl"
    (file "d151/main.tcl" line 1)

Nothing about the script changed. What changed is the current working directory.

source lib/util.tcl is a relative path, and Tcl resolves a relative path against the directory you are in, not the directory the script lives in. From the script's own directory those are the same, which is why it works while you develop it and fails the first time someone runs it from anywhere else — from their home directory, from a Makefile, from a scheduler.

6. Finding the script's own directory

The fix is to stop guessing and ask Tcl where the running script is. Three commands, each adding one layer:

puts "  pwd                : [pwd]"
puts "  info script        : [info script]"
puts "  normalized dir     : [file dirname [file normalize [info script]]]"

Run from the script's directory:

  pwd                : /private/tmp/rt15
  info script        : wa.tcl
  normalized dir     : /private/tmp/rt15

Run from the parent directory as tclsh rt15/wa.tcl:

  pwd                : /private/tmp
  info script        : rt15/wa.tcl
  normalized dir     : /private/tmp/rt15

Compare the two runs line by line. pwd is different. info script is different. normalized dir is the same. That last line is the only one you can build a program on.

Build it up one piece at a time:

• info script is the path of the file being evaluated, exactly as it was given on the command line — so it is wa.tcl in one case and rt15/wa.tcl in the other. On its own it is not enough.
• file normalize turns that into a full absolute path, resolving ., .. and symbolic links.
• file dirname then strips the filename, leaving the directory the script lives in.

The result is the same from both directories. That is the property you need.

This is the idiom to memorise, and it is what the rest of Module 15 uses:

set here [file dirname [file normalize [info script]]]
source [file join $here lib util.tcl]

Two lines, and the program can be run from anywhere.

7. What belongs in a library file

Everything above points at one rule: a library file should define things and do nothing else.

Concretely, a file you intend to source should contain:

• proc definitions,
• namespace eval blocks that declare variable defaults,
• nothing that prints, reads input, opens files, or exits.

The reason is simple. Sourcing a file runs it. If running it does work, then you cannot load the procedures without also doing the work — which makes the file impossible to test, impossible to reuse from a second program, and surprising to anyone who sources it to see what is in it.

Chapter 15.7 builds a real two-directory layout on this rule and shows what it buys you. For now the test is: if I source this file and nothing happens, it is a library.

8. Exercises

Exercise 1. In one sentence, what does source actually do?

Exercise 2. A sourced file does set timeout 30. You source it from inside a procedure. Is timeout available after the procedure returns? Is a proc from the same file?

Exercise 3. source lib/util.tcl works for you and fails for a colleague running the same script. What is the most likely reason?

Exercise 4. Write the two lines that source lib/util.tcl relative to the script file rather than the working directory.

Exercise 5. A sourced file raises an error on its tenth line. Which of its procedures are defined afterwards?

Exercise 6. Why should a library file not print anything when it is sourced?