tcltest man page on OpenMandriva

Man page or keyword search:  
man Server   8135 pages
apropos Keyword Search (all sections)
Output format
OpenMandriva logo
[printable version]

tcltest(n)		     Tcl Bundled Packages		    tcltest(n)

______________________________________________________________________________

NAME
       tcltest - Test harness support code and utilities

SYNOPSIS
       package require tcltest ?2.3?

       tcltest::test name description ?-option value ...?
       tcltest::test name description ?constraints? body result

       tcltest::loadTestedCommands
       tcltest::makeDirectory name ?directory?
       tcltest::removeDirectory name ?directory?
       tcltest::makeFile contents name ?directory?
       tcltest::removeFile name ?directory?
       tcltest::viewFile name ?directory?
       tcltest::cleanupTests ?runningMultipleTests?
       tcltest::runAllTests

       tcltest::configure
       tcltest::configure -option
       tcltest::configure -option value ?-option value ...?
       tcltest::customMatch mode command
       tcltest::testConstraint constraint ?value?
       tcltest::outputChannel ?channelID?
       tcltest::errorChannel ?channelID?
       tcltest::interpreter ?interp?

       tcltest::debug ?level?
       tcltest::errorFile ?filename?
       tcltest::limitConstraints ?boolean?
       tcltest::loadFile ?filename?
       tcltest::loadScript ?script?
       tcltest::match ?patternList?
       tcltest::matchDirectories ?patternList?
       tcltest::matchFiles ?patternList?
       tcltest::outputFile ?filename?
       tcltest::preserveCore ?level?
       tcltest::singleProcess ?boolean?
       tcltest::skip ?patternList?
       tcltest::skipDirectories ?patternList?
       tcltest::skipFiles ?patternList?
       tcltest::temporaryDirectory ?directory?
       tcltest::testsDirectory ?directory?
       tcltest::verbose ?level?

       tcltest::test name description optionList
       tcltest::bytestring string
       tcltest::normalizeMsg msg
       tcltest::normalizePath pathVar
       tcltest::workingDirectory ?dir?
_________________________________________________________________

DESCRIPTION
       The  tcltest  package  provides	several utility commands useful in the
       construction of test suites for code instrumented to be run by  evalua‐
       tion of Tcl commands.  Notably the built-in commands of the Tcl library
       itself are tested by a test suite using the tcltest package.

       All the commands provided by the tcltest package	 are  defined  in  and
       exported	 from  the  ::tcltest  namespace, as indicated in the SYNOPSIS
       above.  In the following sections, all commands will  be	 described  by
       their simple names, in the interest of brevity.

       The  central  command  of tcltest is test that defines and runs a test.
       Testing with test involves evaluation of a Tcl script and comparing the
       result  to an expected result, as configured and controlled by a number
       of options.  Several other commands provided by tcltest govern the con‐
       figuration  of  test and the collection of many test commands into test
       suites.

       See CREATING TEST SUITES WITH TCLTEST below for an extended example  of
       how to use the commands of tcltest to produce test suites for your Tcl-
       enabled code.

COMMANDS
       test name description ?-option value ...?
	      Defines and possibly runs a test with the name name and descrip‐
	      tion  description.   The name and description of a test are used
	      in messages reported by test during the test, as	configured  by
	      the options of tcltest.  The remaining option value arguments to
	      test define the test, including the scripts to run,  the	condi‐
	      tions  under  which  to  run  them, the expected result, and the
	      means by which the expected and actual results  should  be  com‐
	      pared.   See TESTS below for a complete description of the valid
	      options and how they define a test.  The test command returns an
	      empty string.

       test name description ?constraints? body result
	      This form of test is provided to support test suites written for
	      version 1 of the tcltest package, and also a  simpler  interface
	      for  a  common  usage.  It is the same as “test name description
	      -constraints constraints -body body -result result”.  All	 other
	      options  to test take their default values.  When constraints is
	      omitted, this form of test can be distinguished from  the	 first
	      because all options begin with “-”.

       loadTestedCommands
	      Evaluates	 in  the caller's context the script specified by con‐
	      figure -load or configure -loadfile.  Returns the result of that
	      script  evaluation,  including  any  error raised by the script.
	      Use this command and the related configuration options  to  pro‐
	      vide  the	 commands  to be tested to the interpreter running the
	      test suite.

       makeFile contents name ?directory?
	      Creates a file named name relative to  directory	directory  and
	      write  contents to that file using the encoding encoding system.
	      If contents does not end with  a	newline,  a  newline  will  be
	      appended	so  that  the file named name does end with a newline.
	      Because the system encoding is used, this command is only	 suit‐
	      able  for	 making	 text  files.  The file will be removed by the
	      next evaluation of cleanupTests, unless it is removed by remove‐
	      File  first.   The  default  value of directory is the directory
	      configure -tmpdir.  Returns the full path of the	file  created.
	      Use this command to create any text file required by a test with
	      contents as needed.

       removeFile name ?directory?
	      Forces the file referenced by name to  be	 removed.   This  file
	      name  should  be	relative  to directory.	  The default value of
	      directory is the directory configure -tmpdir.  Returns an	 empty
	      string.  Use this command to delete files created by makeFile.

       makeDirectory name ?directory?
	      Creates  a directory named name relative to directory directory.
	      The  directory  will  be	removed	 by  the  next	evaluation  of
	      cleanupTests,  unless  it	 is  removed by removeDirectory first.
	      The default  value  of  directory	 is  the  directory  configure
	      -tmpdir.	 Returns  the full path of the directory created.  Use
	      this command to create any  directories  that  are  required  to
	      exist by a test.

       removeDirectory name ?directory?
	      Forces  the  directory  referenced  by  name to be removed. This
	      directory should be relative to directory.  The default value of
	      directory	 is the directory configure -tmpdir.  Returns an empty
	      string.  Use this command to delete any directories  created  by
	      makeDirectory.

       viewFile file ?directory?
	      Returns the contents of file, except for any final newline, just
	      as read -nonewline would return.	This file name should be rela‐
	      tive to directory.  The default value of directory is the direc‐
	      tory configure -tmpdir.  Use this command as a convenient way to
	      turn  the contents of a file generated by a test into the result
	      of that test for matching against an expected result.  The  con‐
	      tents  of	 the  file  are read using the system encoding, so its
	      usefulness is limited to text files.

       cleanupTests
	      Intended to clean up and summarize after several tests have been
	      run.   Typically	called	once  per test file, at the end of the
	      file after all tests have been completed.	 For  best  effective‐
	      ness,  be	 sure  that  the  cleanupTests is evaluated even if an
	      error occurs earlier in the test file evaluation.

	      Prints statistics about the tests run  and  removes  files  that
	      were  created  by	 makeDirectory	and  makeFile  since  the last
	      cleanupTests.  Names of files and directories in	the  directory
	      configure	 -tmpdir  created since the last cleanupTests, but not
	      created by makeFile or makeDirectory are printed to  outputChan‐
	      nel.  This command also restores the original shell environment,
	      as described by the global env array. Returns an empty string.

       runAllTests
	      This is a master command meant to run an entire suite of	tests,
	      spanning	multiple  files and/or directories, as governed by the
	      configurable options of tcltest.	See RUNNING  ALL  TESTS	 below
	      for  a complete description of the many variations possible with
	      runAllTests.

   CONFIGURATION COMMANDS
       configure
	      Returns the list of configurable options supported  by  tcltest.
	      See  CONFIGURABLE	 OPTIONS  below	 for the full list of options,
	      their valid values, and their effect on tcltest operations.

       configure option
	      Returns the current value of the supported  configurable	option
	      option.	Raises	an  error if option is not a supported config‐
	      urable option.

       configure option value ?-option value ...?
	      Sets the value of each configurable option option to the	corre‐
	      sponding value value, in order.  Raises an error if an option is
	      not a supported configurable option, or if value is not a	 valid
	      value  for  the  corresponding option, or if a value is not pro‐
	      vided.  When an error is raised, the operation of	 configure  is
	      halted, and subsequent option value arguments are not processed.

	      If  the  environment variable ::env(TCLTEST_OPTIONS) exists when
	      the tcltest package is loaded (by package require tcltest)  then
	      its  value is taken as a list of arguments to pass to configure.
	      This allows the default values of the configuration  options  to
	      be set by the environment.

       customMatch mode script
	      Registers	 mode  as  a  new  legal value of the -match option to
	      test.  When the -match mode option is passed to test, the script
	      script  will be evaluated to compare the actual result of evalu‐
	      ating the body of the test to the expected result.   To  perform
	      the  match,  the	script is completed with two additional words,
	      the expected result, and the actual result,  and	the  completed
	      script  is  evaluated  in	 the  global namespace.	 The completed
	      script is expected to return a boolean value indicating  whether
	      or  not  the results match.  The built-in matching modes of test
	      are exact, glob, and regexp.

       testConstraint constraint ?boolean?
	      Sets or returns the boolean value associated with the named con‐
	      straint.	See TEST CONSTRAINTS below for more information.

       interpreter ?executableName?
	      Sets  or	returns	 the  name  of	the executable to be execed by
	      runAllTests to run each test file when configure -singleproc  is
	      false.   The  default  value  for interpreter is the name of the
	      currently running program as returned by info nameofexecutable.

       outputChannel ?channelID?
	      Sets or returns the output channel ID.  This defaults to stdout.
	      Any test that prints test related output should send that output
	      to outputChannel rather than letting that output default to std‐
	      out.

       errorChannel ?channelID?
	      Sets  or returns the error channel ID.  This defaults to stderr.
	      Any test that prints error messages should send that  output  to
	      errorChannel rather than printing directly to stderr.

   SHORTCUT CONFIGURATION COMMANDS
       debug ?level?
	      Same as “configure -debug ?level?”.

       errorFile ?filename?
	      Same as “configure -errfile ?filename?”.

       limitConstraints ?boolean?
	      Same as “configure -limitconstraints ?boolean?”.

       loadFile ?filename?
	      Same as “configure -loadfile ?filename?”.

       loadScript ?script?
	      Same as “configure -load ?script?”.

       match ?patternList?
	      Same as “configure -match ?patternList?”.

       matchDirectories ?patternList?
	      Same as “configure -relateddir ?patternList?”.

       matchFiles ?patternList?
	      Same as “configure -file ?patternList?”.

       outputFile ?filename?
	      Same as “configure -outfile ?filename?”.

       preserveCore ?level?
	      Same as “configure -preservecore ?level?”.

       singleProcess ?boolean?
	      Same as “configure -singleproc ?boolean?”.

       skip ?patternList?
	      Same as “configure -skip ?patternList?”.

       skipDirectories ?patternList?
	      Same as “configure -asidefromdir ?patternList?”.

       skipFiles ?patternList?
	      Same as “configure -notfile ?patternList?”.

       temporaryDirectory ?directory?
	      Same as “configure -tmpdir ?directory?”.

       testsDirectory ?directory?
	      Same as “configure -testdir ?directory?”.

       verbose ?level?
	      Same as “configure -verbose ?level?”.

   OTHER COMMANDS
       The  remaining  commands	 provided  by tcltest have better alternatives
       provided by tcltest or Tcl itself.  They are retained to support exist‐
       ing test suites, but should be avoided in new code.

       test name description optionList
	      This  form  of  test was provided to enable passing many options
	      spanning several lines to test as a single  argument  quoted  by
	      braces,  rather  than  needing  to  backslash quote the newlines
	      between arguments to test.  The optionList argument is  expected
	      to be a list with an even number of elements representing option
	      and value arguments to pass to test.  However, these values  are
	      not  passed  directly,  as  in  the  alternate  forms of switch.
	      Instead, this form makes an  unfortunate	attempt	 to  overthrow
	      Tcl's  substitution rules by performing substitutions on some of
	      the list elements as an attempt to implement a “do what I	 mean”
	      interpretation  of  a  brace-enclosed  “block”.	The  result is
	      nearly impossible to document clearly, and for that reason  this
	      form  is	not  recommended.   See	 the examples in CREATING TEST
	      SUITES WITH TCLTEST below to see that this form  is  really  not
	      necessary	 to avoid backslash-quoted newlines.  If you insist on
	      using this form, examine the source code of tcltest if you  want
	      to  know	the  substitution  details,  or just enclose the third
	      through last argument to test in braces and hope for the best.

       workingDirectory ?directoryName?
	      Sets or returns the current  working  directory  when  the  test
	      suite is running.	 The default value for workingDirectory is the
	      directory in which the test suite was launched.	The  Tcl  com‐
	      mands cd and pwd are sufficient replacements.

       normalizeMsg msg
	      Returns  the  result  of removing the “extra” newlines from msg,
	      where “extra” is rather imprecise.  Tcl offers plenty of	string
	      processing  commands  to modify strings as you wish, and custom‐
	      Match allows flexible matching of actual and expected results.

       normalizePath pathVar
	      Resolves symlinks in a path, thus creating a path without inter‐
	      nal redirection.	It is assumed that pathVar is absolute.	 path‐
	      Var is modified in place.	 The Tcl command file normalize	 is  a
	      sufficient replacement.

       bytestring string
	      Construct	 a  string  that consists of the requested sequence of
	      bytes, as opposed to a string of properly formed	UTF-8  charac‐
	      ters using the value supplied in string.	This allows the tester
	      to create denormalized or improperly formed strings to pass to C
	      procedures  that	are  supposed  to accept strings with embedded
	      NULL types and confirm that a string result has a	 certain  pat‐
	      tern  of	bytes.	 This is exactly equivalent to the Tcl command
	      encoding convertfrom identity.

TESTS
       The test command is the heart of the tcltest  package.	Its  essential
       function	 is  to	 evaluate  a Tcl script and compare the result with an
       expected result.	 The options of test define the test script, the envi‐
       ronment	in which to evaluate it, the expected result, and how the com‐
       pare the actual result to  the  expected	 result.   Some	 configuration
       options of tcltest also influence how test operates.

       The valid options for test are summarized:

	      test name description
		      ?-constraints keywordList|expression?
		      ?-setup setupScript?
		      ?-body testScript?
		      ?-cleanup cleanupScript?
		      ?-result expectedAnswer?
		      ?-output expectedOutput?
		      ?-errorOutput expectedError?
		      ?-returnCodes codeList?
		      ?-match mode?

       The  name  may  be  any	string.	  It  is conventional to choose a name
       according to the pattern:

	      target-majorNum.minorNum

       For white-box (regression) tests, the target should be the name of  the
       C  function  or	Tcl  procedure being tested.  For black-box tests, the
       target should be the name of the feature being  tested.	 Some  conven‐
       tions  call  for	 the  names of black-box tests to have the suffix _bb.
       Related tests should share a major number.  As a test suite evolves, it
       is  best	 to have the same test name continue to correspond to the same
       test, so that it remains meaningful to say things  like	“Test  foo-1.3
       passed in all releases up to 3.4, but began failing in release 3.5.”

       During  evaluation  of  test, the name will be compared to the lists of
       string matching patterns returned by configure  -match,	and  configure
       -skip.	The  test will be run only if name matches any of the patterns
       from configure -match and matches none of the patterns  from  configure
       -skip.

       The description should be a short textual description of the test.  The
       description is included in output produced by the test, typically  test
       failure	messages.   Good description values should briefly explain the
       purpose of the test to users of a test suite.  The name of a Tcl	 or  C
       function being tested should be included in the description for regres‐
       sion tests.  If the test case exists to reproduce a  bug,  include  the
       bug ID in the description.

       Valid attributes and associated values are:

       -constraints keywordList|expression
	      The  optional  -constraints attribute can be list of one or more
	      keywords or an expression.  If the -constraints value is a  list
	      of keywords, each of these keywords should be the name of a con‐
	      straint defined by a call to  testConstraint.   If  any  of  the
	      listed  constraints  is  false  or  does	not exist, the test is
	      skipped.	If the	-constraints  value  is	 an  expression,  that
	      expression  is  evaluated.  If the expression evaluates to true,
	      then the test is run.  Note that the expression  form  of	 -con‐
	      straints	may  interfere	with  the operation of configure -con‐
	      straints and configure  -limitconstraints,  and  is  not	recom‐
	      mended.	Appropriate  constraints  should be added to any tests
	      that should not always be run.  That is, conditional  evaluation
	      of a test should be accomplished by the -constraints option, not
	      by conditional evaluation of test.  In that way, the same number
	      of  tests are always reported by the test suite, though the num‐
	      ber skipped may change based on the  testing  environment.   The
	      default  value is an empty list.	See TEST CONSTRAINTS below for
	      a list of built-in constraints and information  on  how  to  add
	      your own constraints.

       -setup script
	      The  optional  -setup  attribute indicates a script that will be
	      run before the script indicated  by  the	-body  attribute.   If
	      evaluation  of  script raises an error, the test will fail.  The
	      default value is an empty script.

       -body script
	      The -body attribute indicates the script to run to carry out the
	      test,  which  must  return a result that can be checked for cor‐
	      rectness.	 If evaluation of script raises	 an  error,  the  test
	      will  fail (unless the -returnCodes option is used to state that
	      an error is expected).  The default value is an empty script.

       -cleanup script
	      The optional -cleanup attribute indicates a script that will  be
	      run after the script indicated by the -body attribute.  If eval‐
	      uation of script raises an  error,  the  test  will  fail.   The
	      default value is an empty script.

       -match mode
	      The -match attribute determines how expected answers supplied by
	      -result, -output, and -errorOutput are compared.	 Valid	values
	      for  mode are regexp, glob, exact, and any value registered by a
	      prior call to customMatch.  The default value is exact.

       -result expectedValue
	      The -result attribute supplies the expectedValue	against	 which
	      the return value from script will be compared. The default value
	      is an empty string.

       -output expectedValue
	      The -output attribute supplies the expectedValue	against	 which
	      any  output sent to stdout or outputChannel during evaluation of
	      the script(s) will be compared.  Note that only  output  printed
	      using  the global puts command is used for comparison.  If -out‐
	      put is not specified, output sent to stdout and outputChannel is
	      not processed for comparison.

       -errorOutput expectedValue
	      The  -errorOutput	 attribute  supplies the expectedValue against
	      which any output sent to stderr or errorChannel  during  evalua‐
	      tion  of	the  script(s) will be compared. Note that only output
	      printed using the global puts command is	used  for  comparison.
	      If  -errorOutput	is  not	 specified,  output sent to stderr and
	      errorChannel is not processed for comparison.

       -returnCodes expectedCodeList
	      The optional -returnCodes attribute supplies expectedCodeList, a
	      list of return codes that may be accepted from evaluation of the
	      -body script.  If evaluation of the -body script returns a  code
	      not  in  the expectedCodeList, the test fails.  All return codes
	      known to return, in both numeric and  symbolic  form,  including
	      extended	return codes, are acceptable elements in the expected‐
	      CodeList.	 Default value is “ok return”.

       To pass, a test must  successfully  evaluate  its  -setup,  -body,  and
       -cleanup	 scripts.   The return code of the -body script and its result
       must match expected values, and if specified,  output  and  error  data
       from  the test must match expected -output and -errorOutput values.  If
       any of these conditions are not met, then the test  fails.   Note  that
       all scripts are evaluated in the context of the caller of test.

       As  long	 as  test is called with valid syntax and legal values for all
       attributes, it will not raise an	 error.	  Test	failures  are  instead
       reported	 as  output written to outputChannel.  In default operation, a
       successful test produces no output.  The output	messages  produced  by
       test  are  controlled  by the configure -verbose option as described in
       CONFIGURABLE OPTIONS below.  Any output produced by  the	 test  scripts
       themselves should be produced using puts to outputChannel or errorChan‐
       nel, so that users of the test suite may easily capture output with the
       configure  -outfile  and	 configure  -errfile  options, and so that the
       -output and -errorOutput attributes work properly.

   TEST CONSTRAINTS
       Constraints are used to determine whether  or  not  a  test  should  be
       skipped.	  Each	constraint  has a name, which may be any string, and a
       boolean value.  Each test has a -constraints value which is a  list  of
       constraint  names.   There  are	two modes of constraint control.  Most
       frequently, the default mode is used, indicated by a setting of config‐
       ure  -limitconstraints  to  false.   The test will run only if all con‐
       straints in the list are true-valued.  Thus, the -constraints option of
       test  is	 a  convenient, symbolic way to define any conditions required
       for the test to be possible or meaningful.  For example,	 a  test  with
       -constraints  unix  will	 only  be  run if the constraint unix is true,
       which indicates the test suite is being run on a Unix platform.

       Each test should include whatever -constraints  are  required  to  con‐
       strain  it to run only where appropriate.  Several constraints are pre-
       defined in the tcltest package,	listed	below.	 The  registration  of
       user-defined  constraints  is  performed by the testConstraint command.
       User-defined constraints may appear within a test file, or  within  the
       script specified by the configure -load or configure -loadfile options.

       The following is a list of constraints pre-defined by the tcltest pack‐
       age itself:

       singleTestInterp
	      This test can only be run if all test files are sourced  into  a
	      single interpreter.

       unix   This test can only be run on any Unix platform.

       win    This test can only be run on any Windows platform.

       nt     This test can only be run on any Windows NT platform.

       95     This test can only be run on any Windows 95 platform.

       98     This test can only be run on any Windows 98 platform.

       mac    This test can only be run on any Mac platform.

       unixOrWin
	      This test can only be run on a Unix or Windows platform.

       macOrWin
	      This test can only be run on a Mac or Windows platform.

       macOrUnix
	      This test can only be run on a Mac or Unix platform.

       tempNotWin
	      This  test can not be run on Windows.  This flag is used to tem‐
	      porarily disable a test.

       tempNotMac
	      This test can not be run on a Mac.  This flag is used to	tempo‐
	      rarily disable a test.

       unixCrash
	      This  test  crashes  if it is run on Unix.  This flag is used to
	      temporarily disable a test.

       winCrash
	      This test crashes if it is run on Windows.  This flag is used to
	      temporarily disable a test.

       macCrash
	      This  test  crashes if it is run on a Mac.  This flag is used to
	      temporarily disable a test.

       emptyTest
	      This test is empty, and so not worth running, but it remains  as
	      a	 place-holder  for  a  test to be written in the future.  This
	      constraint has value false to cause tests to be  skipped	unless
	      the user specifies otherwise.

       knownBug
	      This  test  is known to fail and the bug is not yet fixed.  This
	      constraint has value false to cause tests to be  skipped	unless
	      the user specifies otherwise.

       nonPortable
	      This test can only be run in some known development environment.
	      Some tests are inherently non-portable because  they  depend  on
	      things  like word length, file system configuration, window man‐
	      ager, etc.  This constraint has value false to cause tests to be
	      skipped unless the user specifies otherwise.

       userInteraction
	      This  test  requires interaction from the user.  This constraint
	      has value false to causes tests to be skipped  unless  the  user
	      specifies otherwise.

       interactive
	      This  test  can only be run in if the interpreter is in interac‐
	      tive mode (when the global tcl_interactive variable  is  set  to
	      1).

       nonBlockFiles
	      This  test  can  only  be run if platform supports setting files
	      into nonblocking mode.

       asyncPipeClose
	      This test can only be run if platform supports async  flush  and
	      async close on a pipe.

       unixExecs
	      This  test  can  only be run if this machine has Unix-style com‐
	      mands cat, echo, sh, wc, rm, sleep, fgrep, ps, chmod, and	 mkdir
	      available.

       hasIsoLocale
	      This test can only be run if can switch to an ISO locale.

       root   This test can only run if Unix user is root.

       notRoot
	      This test can only run if Unix user is not root.

       eformat
	      This  test  can only run if app has a working version of sprintf
	      with respect to the “e” format of floating-point numbers.

       stdio  This test can only be run if interpreter	can  be	 opened	 as  a
	      pipe.

       The  alternative	 mode of constraint control is enabled by setting con‐
       figure -limitconstraints to true.  With that configuration setting, all
       existing	 constraints  other than those in the constraint list returned
       by configure -constraints are set to false.  When the value of  config‐
       ure  -constraints  is  set, all those constraints are set to true.  The
       effect is that when both options configure -constraints	and  configure
       -limitconstraints  are  in  use,	 only  those tests including only con‐
       straints from the configure -constraints list are run; all  others  are
       skipped.	 For example, one might set up a configuration with

	      configure -constraints knownBug \
			-limitconstraints true \
			-verbose pass

       to  run	exactly	 those	tests  that  exercise known bugs, and discover
       whether any of them pass, indicating the bug had been fixed.

   RUNNING ALL TESTS
       The single command runAllTests is  evaluated  to	 run  an  entire  test
       suite,  spanning many files and directories.  The configuration options
       of tcltest control the precise  operations.   The  runAllTests  command
       begins by printing a summary of its configuration to outputChannel.

       Test files to be evaluated are sought in the directory configure -test‐
       dir.  The list of files in that directory that match any	 of  the  pat‐
       terns  in  configure  -file and match none of the patterns in configure
       -notfile is generated and sorted.  Then each file will be evaluated  in
       turn.  If configure -singleproc is true, then each file will be sourced
       in the caller's context.	 If it is false, then a	 copy  of  interpreter
       will  be	 exec'd to evaluate each file.	The multi-process operation is
       useful when testing can cause errors so severe that  a  process	termi‐
       nates.  Although such an error may terminate a child process evaluating
       one file, the master process can continue with the  rest	 of  the  test
       suite.  In multi-process operation, the configuration of tcltest in the
       master process is passed to the child processes as command  line	 argu‐
       ments,  with the exception of configure -outfile.  The runAllTests com‐
       mand in the master process collects all output from the child processes
       and  collates  their  results  into  one master report.	Any reports of
       individual test failures, or messages requested by a configure -verbose
       setting are passed directly on to outputChannel by the master process.

       After  evaluating  all selected test files, a summary of the results is
       printed to outputChannel.  The summary includes	the  total  number  of
       tests  evaluated,  broken  down	into  those skipped, those passed, and
       those failed.  The summary also notes the number	 of  files  evaluated,
       and the names of any files with failing tests or errors.	 A list of the
       constraints that caused tests to be skipped, and the  number  of	 tests
       skipped	for  each  is  also printed.  Also, messages are printed if it
       appears that evaluation of a test file has caused any  temporary	 files
       to be left behind in configure -tmpdir.

       Having  completed  and  summarized all selected test files, runAllTests
       then recursively acts on subdirectories	of  configure  -testdir.   All
       subdirectories  that match any of the patterns in configure -relateddir
       and do not match any of the patterns  in	 configure  -asidefromdir  are
       examined.   If  a  file	named all.tcl is found in such a directory, it
       will be sourced in the caller's context.	 Whether or  not  an  examined
       directory contains an all.tcl file, its subdirectories are also scanned
       against the configure -relateddir and configure -asidefromdir patterns.
       In  this	 way,  many directories in a directory tree can have all their
       test files evaluated by a single runAllTests command.

CONFIGURABLE OPTIONS
       The configure command is used to set and query the configurable options
       of tcltest.  The valid options are:

       -singleproc boolean
	      Controls	whether	 or not runAllTests spawns a child process for
	      each test file.  No spawning  when  boolean  is  true.   Default
	      value is false.

       -debug level
	      Sets  the	 debug level to level, an integer value indicating how
	      much debugging information should be printed  to	stdout.	  Note
	      that  debug  messages  always  go	 to stdout, independent of the
	      value of configure -outfile.  Default value is  0.   Levels  are
	      defined as:

	      0	  Do not display any debug information.

	      1	  Display  information	regarding  whether  a  test is skipped
		  because it does not match any of the tests that were	speci‐
		  fied	using  by  configure -match (userSpecifiedNonMatch) or
		  matches any of the tests specified by configure -skip (user‐
		  SpecifiedSkip).   Also print warnings about possible lack of
		  cleanup or balance in test files.  Also print warnings about
		  any re-use of test names.

	      2	  Display the flag array parsed by the command line processor,
		  the contents of the global env array, and  all  user-defined
		  variables  that  exist  in the current namespace as they are
		  used.

	      3	  Display information regarding what individual procs  in  the
		  test harness are doing.

       -verbose level
	      Sets  the	 type  of output verbosity desired to level, a list of
	      zero or more of the elements body, pass, skip, start, error  and
	      line.  Default value is “body error”.  Levels are defined as:

	      body (b)
		     Display the body of failed tests

	      pass (p)
		     Print output when a test passes

	      skip (s)
		     Print output when a test is skipped

	      start (t)
		     Print output whenever a test starts

	      error (e)
		     Print errorInfo and errorCode, if they exist, when a test
		     return code does not match its expected return code

	      line (l)
		     Print source file line information of failed tests

	      The single letter abbreviations noted above are also  recognized
	      so  that “configure -verbose pt” is the same as “configure -ver‐
	      bose {pass start}”.

       -preservecore level
	      Sets the core preservation level to level.   This	 level	deter‐
	      mines how stringent checks for core files are.  Default value is
	      0.  Levels are defined as:

	      0	     No checking — do not check for core files at the  end  of
		     each  test	 command, but do check for them in runAllTests
		     after all test files have been evaluated.

	      1	     Also check for core files at the end of  each  test  com‐
		     mand.

	      2	     Check  for	 core  files at all times described above, and
		     save a copy of  each  core	 file  produced	 in  configure
		     -tmpdir.

       -limitconstraints boolean
	      Sets  the	 mode by which test honors constraints as described in
	      TESTS above.  Default value is false.

       -constraints list
	      Sets all the constraints in list to true.	 Also used in combina‐
	      tion  with configure -limitconstraints true to control an alter‐
	      native constraint mode as described  in  TESTS  above.   Default
	      value is an empty list.

       -tmpdir directory
	      Sets  the temporary directory to be used by makeFile, makeDirec‐
	      tory, viewFile, removeFile, and removeDirectory as  the  default
	      directory	 where temporary files and directories created by test
	      files should be created.	Default value is workingDirectory.

       -testdir directory
	      Sets the directory searched by runAllTests for  test  files  and
	      subdirectories.  Default value is workingDirectory.

       -file patternList
	      Sets  the list of patterns used by runAllTests to determine what
	      test files to evaluate.  Default value is “*.test”.

       -notfile patternList
	      Sets the list of patterns used by runAllTests to determine  what
	      test  files  to  skip.  Default value is “l.*.test”, so that any
	      SCCS lock files are skipped.

       -relateddir patternList
	      Sets the list of patterns used by runAllTests to determine  what
	      subdirectories  to search for an all.tcl file.  Default value is
	      “*”.

       -asidefromdir patternList
	      Sets the list of patterns used by runAllTests to determine  what
	      subdirectories  to  skip	when  searching	 for  an all.tcl file.
	      Default value is an empty list.

       -match patternList
	      Set the list of patterns used by test  to	 determine  whether  a
	      test should be run.  Default value is “*”.

       -skip patternList
	      Set  the	list  of  patterns used by test to determine whether a
	      test should be skipped.  Default value is an empty list.

       -load script
	      Sets a script to be evaluated  by	 loadTestedCommands.   Default
	      value is an empty script.

       -loadfile filename
	      Sets the filename from which to read a script to be evaluated by
	      loadTestedCommands.  This is an alternative to -load.  They can‐
	      not be used together.

       -outfile filename
	      Sets  the file to which all output produced by tcltest should be
	      written.	A file named filename will be opened for writing,  and
	      the resulting channel will be set as the value of outputChannel.

       -errfile filename
	      Sets  the	 file  to  which  all error output produced by tcltest
	      should be written.  A file named filename	 will  be  opened  for
	      writing,	and  the resulting channel will be set as the value of
	      errorChannel.

CREATING TEST SUITES WITH TCLTEST
       The fundamental element of a test suite is the individual test command.
       We begin with several examples.

       [1]    Test of a script that returns normally.

		     test example-1.0 {normal return} {
			 format %s value
		     } value

       [2]    Test  of a script that requires context setup and cleanup.  Note
	      the bracing and indenting style that avoids any  need  for  line
	      continuation.

		     test example-1.1 {test file existence} -setup {
			 set file [makeFile {} test]
		     } -body {
			 file exists $file
		     } -cleanup {
			 removeFile test
		     } -result 1

       [3]    Test of a script that raises an error.

		     test example-1.2 {error return} -body {
			 error message
		     } -returnCodes error -result message

       [4]    Test with a constraint.

		     test example-1.3 {user owns created files} -constraints {
			 unix
		     } -setup {
			 set file [makeFile {} test]
		     } -body {
			 file attributes $file -owner
		     } -cleanup {
			 removeFile test
		     } -result $::tcl_platform(user)

       At  the	next  higher  layer of organization, several test commands are
       gathered together into a single test  file.   Test  files  should  have
       names  with  the “.test” extension, because that is the default pattern
       used by runAllTests to find test files.	It is a good rule of thumb  to
       have  one  test	file for each source code file of your project.	 It is
       good practice to edit the test file and the source code file  together,
       keeping tests synchronized with code changes.

       Most  of	 the  code  in the test file should be the test commands.  Use
       constraints to skip tests, rather than conditional evaluation of test.

       [5]    Recommended system for writing  conditional  tests,  using  con‐
	      straints to guard:

		     testConstraint X [expr $myRequirement]
		     test goodConditionalTest {} X {
			 # body
		     } result

       [6]    Discouraged  system  for	writing conditional tests, using if to
	      guard:

		     if $myRequirement {
			 test badConditionalTest {} {
			     #body
			 } result
		     }

       Use the -setup and -cleanup options to establish and release  all  con‐
       text  requirements of the test body.  Do not make tests depend on prior
       tests in the file.  Those prior tests might  be	skipped.   If  several
       consecutive  tests  require the same context, the appropriate setup and
       cleanup scripts may be stored in variable for  passing  to  each	 tests
       -setup and -cleanup options.  This is a better solution than performing
       setup outside of test commands, because the setup will only be done  if
       necessary,  and any errors during setup will be reported, and not cause
       the test file to abort.

       A test file should be able to be combined with other test files and not
       interfere with them, even when configure -singleproc 1 causes all files
       to be evaluated in a common interpreter.	 A simple way to achieve  this
       is  to  have  your  tests  define all their commands and variables in a
       namespace that is deleted when the test file evaluation is complete.  A
       good namespace to use is a child namespace test of the namespace of the
       module you are testing.

       A test file should also be able to be evaluated directly as  a  script,
       not depending on being called by a master runAllTests.  This means that
       each test file should process command line arguments to give the tester
       all the configuration control that tcltest provides.

       After  all  tests  in  a	 test file, the command cleanupTests should be
       called.

       [7]    Here is a sketch	of  a  sample  test  file  illustrating	 those
	      points:

		     package require tcltest 2.2
		     eval ::tcltest::configure $argv
		     package require example
		     namespace eval ::example::test {
			 namespace import ::tcltest::*
			 testConstraint X [expr {...}]
			 variable SETUP {#common setup code}
			 variable CLEANUP {#common cleanup code}
			 test example-1 {} -setup $SETUP -body {
			     # First test
			 } -cleanup $CLEANUP -result {...}
			 test example-2 {} -constraints X -setup $SETUP -body {
			     # Second test; constrained
			 } -cleanup $CLEANUP -result {...}
			 test example-3 {} {
			     # Third test; no context required
			 } {...}
			 cleanupTests
		     }
		     namespace delete ::example::test

       The next level of organization is a full test suite, made up of several
       test files.  One script is used to control the entire suite.  The basic
       function	 of  this script is to call runAllTests after doing any neces‐
       sary setup.  This script is usually named all.tcl because that  is  the
       default	name  used  by runAllTests when combining multiple test suites
       into one testing run.

       [8]    Here is a sketch of a sample test suite master script:

		     package require Tcl 8.4
		     package require tcltest 2.2
		     package require example
		     ::tcltest::configure -testdir \
			     [file dirname [file normalize [info script]]]
		     eval ::tcltest::configure $argv
		     ::tcltest::runAllTests

COMPATIBILITY
       A number of commands and variables in the ::tcltest namespace  provided
       by earlier releases of tcltest have not been documented here.  They are
       no longer part of the supported public interface of tcltest and	should
       not be used in new test suites.	However, to continue to support exist‐
       ing test suites written to the older interface specifications, many  of
       those  deprecated  commands  and	 variables  still work as before.  For
       example, in many circumstances, configure will be automatically	called
       shortly	after package require tcltest 2.1 succeeds with arguments from
       the variable ::argv.  This is to support test suites that depend on the
       old  behavior  that  tcltest  was automatically configured from command
       line arguments.	New test files should not depend on this,  but	should
       explicitly include

	      eval ::tcltest::configure $::argv

       or

	      ::tcltest::configure {*}$::argv

       to establish a configuration from command line arguments.

KNOWN ISSUES
       There  are two known issues related to nested evaluations of test.  The
       first issue relates to the stack level in which test scripts  are  exe‐
       cuted.	Tests  nested  within  other tests may be executed at the same
       stack level as the outermost test.  For example, in the following code:

	      test level-1.1 {level 1} {
		  -body {
		      test level-2.1 {level 2} {
		      }
		  }
	      }

       any script executed in level-2.1 may be	executed  at  the  same	 stack
       level as the script defined for level-1.1.

       In  addition,  while  two  tests	 have  been  run, results will only be
       reported by cleanupTests for tests at the same level as test level-1.1.
       However,	 test  results	for  all  tests run prior to level-1.1 will be
       available when test level-2.1 runs.  What this means is that if you try
       to access the test results for test level-2.1, it will may say that “m”
       tests have run, “n” tests have been skipped, “o” tests have passed  and
       “p” tests have failed, where “m”, “n”, “o”, and “p” refer to tests that
       were run at the same test level as test level-1.1.

       Implementation of output and  error  comparison	in  the	 test  command
       depends	on  usage  of puts in your application code.  Output is inter‐
       cepted by redefining the global puts command  while  the	 defined  test
       script is being run.  Errors thrown by C procedures or printed directly
       from C applications will not be caught by the test command.  Therefore,
       usage  of  the  -output and -errorOutput options to test is useful only
       for pure Tcl applications that use puts to produce output.

KEYWORDS
       test, test harness, test suite

tcltest				      2.3			    tcltest(n)
[top]

List of man pages available for OpenMandriva

Copyright (c) for man pages and the logo by the respective OS vendor.

For those who want to learn more, the polarhome community provides shell access and support.

[legal] [privacy] [GNU] [policy] [cookies] [netiquette] [sponsors] [FAQ]
Tweet
Polarhome, production since 1999.
Member of Polarhome portal.
Based on Fawad Halim's script.
....................................................................
Vote for polarhome
Free Shell Accounts :: the biggest list on the net