etc/autotest/README

   1 List of files
   2 -------------
   3
   4 - titan_builder.sh: Wrapper script for titan_builder.py.  It sets some basic
   5                     environment variables for platforms with tools installed
   6                     to non-standard locations.
   7 - titan_builder.py: Build process control functions.  It shouldn't be run
   8                     directly, use the wrapper script instead.
   9 - titan_builder_cfg.py: The configuration file.  It contains the build
  10                         configurations, build slaves etc.
  11 - product_handler.py: Helper functions to build the VOB products.
  12 - titan_publisher.py: Different publishing related functions.
  13 - utils.py: General purpose functions.
  14
  15 Brief overview
  16 --------------
  17
  18 When titan_builder.sh is started, it checks out a build configuration specific
  19 version of TITAN from CVS.  TITAN gets configured according to the build
  20 configuration (e.g. variables in Makefile.cfg).  When it's ready, TITAN is
  21 distributed to each slave attached to the given build configuration.  The
  22 builds are performed sequentially.  All script files are distributed as well
  23 and the script starts executing in slave mode on the slave machine.  The master
  24 and the slave can be the same machines.  In this case, using different build
  25 directories for them is strongly recommended.  When all passes are finished on
  26 the slave machine the master starts the build on the next slave machine etc.,
  27 then the remaining build configurations are built.
  28
  29 After all that, the master collects all logs from the slave machines.  The
  30 slaves can produce three kinds of outputs: CSV, TXT, HTML.  The CSV file is
  31 essential, since the master will consider each slave lost without such a file.
  32 The CSV file is used to generate the e-mail notification message as well.
  33 Finally, the master publishes the logs and cleans things up.  The compiled
  34 sources will remain available on each slave until the next build.
  35
  36 To run VOB product tests the VOB or a similar directory structure of products
  37 should be available on master machine.  SSHFS can be used for this purpose.
  38 The performance test package is not part of the official TITAN test suite,
  39 hence it needs to be distributed for the slaves as well.  Currently it's just
  40 an archive reachable on the master machine.  It's recommended to use
  41 configspec 129 and the TCC_Common VOB.
  42
  43 All shell commands are executed using the `run_cmd()' function.  It has a
  44 timeout parameter to kill the command.
  45
  46 To make sure, that always the latest version of the scripts runs, one can use
  47 a simple script, which checks out the sources from CVS first:
  48
  49 #!/bin/bash
  50
  51 rm -rf *.{py,sh} TTCNv3/etc/autotest
  52 cvs co TTCNv3/etc/autotest && ln -sf TTCNv3/etc/autotest/*.{py,sh} .
  53 ./titan_builder.sh -c tests_on_x86_64_linux,tests_on_solaris,tests_on_i386_solaris,limited_tests_on_ubuntu910 1>/dev/null
  54
  55 Prerequisites
  56 -------------
  57
  58 - Python >= 2.3
  59 - Configured password-less SSH connections between the master and the slave
  60 - For scheduled execution use `cron'
  61
  62 Command line options
  63 --------------------
  64
  65 -c [c1,c2,...]: Select the build configurations to run.  If no build
  66                 configuration is selected, then all configurations will be
  67                 run.  It's a comma separated list.
  68 -d: Print all the available build configurations present in the configuration
  69     file.  Slaves attached to a given build configuration are listed as well.
  70 -h: Display available options.
  71 -s: Start the script in slave mode.  This option should only be used by
  72     titan_builder.py, unless everything is in place.
  73
  74 Configuration
  75 -------------
  76
  77 It's a Python source file, take care of the syntax.
  78
  79 - Common options
  80
  81     buiddir:STR The build directory of the master.
  82     logdir:STR  The logs of the master go here.
  83     htmldir:STR All HTML files will be published here.
  84     vob:STR     The VOB products will be copied from here.
  85     archive:INT Archive the logs after a specified number of days.
  86     cleanup:INT Move the archived logs somewhere else after a specified number of days.
  87     measureperiod:INT Reset scores after a given number of days.
  88     cleanupslave:{slave:STR,dir:STR} The destination of abandoned logs.
  89
  90 - Recipients
  91
  92     {name:STR,address:STR}
  93
  94     The names and e-mail addresses of the recipients.
  95
  96 - Slaves
  97
  98     slaves[name:STR]:{ip:STR,user:STR,configs:LIST}
  99
 100     The IP address, the user's name, the build configurations attached to the
 101     slave.  Currently one build configuration is supported for each slave.
 102
 103 - Product descriptions
 104
 105     products[kind:STR]:{name:STR,action0:BOOL,action1:BOOL,...}
 106
 107     `kind' is always a directory under the VOB's root directory (e.g.
 108     TestPorts, ProtocolModules).  `name' is the product name.  Actions can be
 109     enabled individually.  The predefined actions are the following:
 110
 111     - semantic: `compiler -s'
 112     - translate: `compiler'
 113     - compile: `make'
 114     - run: `make run'
 115
 116 - Build configurations
 117
 118     version:STR        Version of TITAN to use.  It can be a CVS tag or date.
 119                        If it's not set the HEAD will be taken.
 120     license:STR        Location of the license file.
 121     gui:BOOL           The `GUI' part in Makefile.cfg.
 122     jni:BOOL           The `JNI' part in Makefile.cfg.
 123     debug:BOOL         The `DEBUG' part in Makefile.cfg.
 124     compilerflags:STR  The `COMPILERFLAGS' in Makefile.cfg.
 125     ldflags:STR        The `LDFLAGS' in Makefile.cfg.
 126     gccdir:STR         This will affect `CC' and `CXX'.
 127     *cc:STR            Value of `CC' in synch with the previous option.
 128     *cxx:STR           Value of `CXX' in synch with the previous option.
 129     flex:STR           Replace `FLEX'.
 130     perl:STR           Location of the `perl' interpreter.
 131     bison:STR          Replace `BISON'.
 132     jdkdir:STR         Replace `JDKDIR'.
 133     qtdir:STR          Replace `QTDIR'.
 134     xmldir:STR         Replace `XMLDIR'.
 135     openssldir:STR     Replace `OPENSSL_DIR'.
 136     regtest:BOOL       Run regression tests.
 137     perftest:BOOL      Run performance tests.  The location of the testsuite
 138                        must be known, since it's not part of CVS.  It should
 139                        be available for the master locally.
 140     perftestdir:STR    Location of the performance tests.
 141     *cpsmin:INT        Minimum CPS value for performance tests.
 142     *cpsmax:INT        Maximum CPS value for performance tests.
 143     functest:BOOL      Run function tests.
 144     vobtest:BOOL       Run product tests.
 145     *vobtest_logs:BOOL Save logs for product tests.
 146     rt2:BOOL           Run tests with both run-times.
 147     builddir:STR       Everything will be done here.  It should be different
 148                        from the master's.
 149     logdir:STR         Place of the logs.
 150     *pdfdir:STR        Local directory to copy .pdf files from.  If not present
 151                        no *.pdf files will be there.  If it's an empty string
 152                        the *.pdf files will be faked with empty files.
 153     *xsdtests:BOOL     Disable regression tests for `xsd2ttcn'.  It's very time
 154                        consuming.
 155     measure:BOOL       Enable `quality' measurements.
 156
 157 (`*' is for optional fields.)
 158
 159 Platform specific settings
 160 --------------------------
 161
 162 If some essential tools (e.g. GCC) are installed to non-standard locations on a
 163 given platform, all of these environmental settings can be placed into a
 164 configuration file `${HOME}/.titan_builder'.  This file is always sourced
 165 first, before starting the actual build.  An example file:
 166
 167 #!/bin/bash
 168
 169 # Basic configuration for `bangjohansen' (172.31.21.76).
 170
 171 PATH=${HOME}/apps/bin:/mnt/TTCN/Tools/gcc-3.4.6-sol10/bin:/mnt/TTCN/Tools/binutils-2.17-sol10/bin:/usr/local/bin:/usr/bin:${PATH}
 172 LD_LIBRARY_PATH=${HOME}/apps/lib:/mnt/TTCN/Tools/gcc-3.4.6-sol10/lib:/mnt/TTCN/Tools/binutils-2.17-sol10/lib:/usr/local/lib:/usr/lib:${LD_LIBRARY_PATH}
 173
 174 TTCN3_LICENSE_FILE=${HOME}/.TTCN3/license_2536.dat
 175 CVSROOT=esekits1064.rnd.ki.sw.ericsson.se:/proj/TTCN/cvs_root
 176 CVS_SERVER=/usr/local/bin/cvs
 177 CVS_RSH=ssh
 178 EDITOR=mcedit
 179
 180 export PATH LD_LIBRARY_PATH CVSROOT CVS_SERVER EDITOR TTCN3_LICENSE_FILE CVS_RSH
 181
 182 When something goes wrong
 183 -------------------------
 184
 185 E.g. no e-mail message is received, garbage or missing sections in the e-mail
 186 or on the generated HTML page...
 187
 188 - Possible reasons and possible solutions
 189
 190   - Problems with e-mail system: In case of Postfix check the `mailq'
 191     command.  All messages still in the queue should be printed.  An example
 192     output of the `mailq' command:
 193
 194     -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
 195     5A57D160B02     1696 Fri Oct 16 09:32:05  ferenc.kovacs@ericsson.com
 196     (host mwux020.eth.ericsson.se[159.107.148.18] said: 452 4.4.5
 197     Insufficient disk space; try again later (in reply to MAIL FROM command))
 198     ferenc.kovacs@ericsson.com
 199
 200     To clear all the messages: `postsuper -d ALL'.
 201
 202   - Exception in the Python code: Check the titan_builder.err-log file of the
 203     master and the slaves.  All exceptions should go into this file.  The
 204     other log file titan_builder.log can be useful as well.
 205
 206   - Disk quota exceeded.
 207
 208   - The SSHFS mounted partitions (e.g. used for HTML publishing) get umounted.
 209     The common cause is a segfault:
 210
 211     Dec 12 07:00:50 tcclab1 kernel: sshfs[20289]: segfault at 6507a ip
 212     7fbf99315064 sp 44288ea8 error 4 in libc-2.5.so[7fbf992a2000+139000]
 213
 214     Check /var/log/messages for details.
 215
 216     If SSHFS locks up and the mounted partitions cannot be umounted like:
 217
 218     titanrt@tcclab1:~> fusermount -u ~/public_html
 219     fusermount: failed to unmount /home/titanrt/public_html: Device or resource busy
 220
 221     Simply `killall -9 sshfs' and try to remount the partitions.  If SSHFS
 222     locks up you may not be able to execute any command in the parent
 223     directory of these mount points.  The commands will simply hang.  If it
 224     doesn't work, try the following:
 225
 226     titanrt@tcclab1:~> lsof | grep public_html
 227     lsof: WARNING: can't stat() fuse file system /home/titanrt/public_html
 228     Output information may be incomplete.
 229     bash 2378  titanrt cwd unknown /home/titanrt/public_html/titan_builds (stat: Transport endpoint is not connected)
 230     ssh  13480 titanrt cwd unknown /home/titanrt/public_html/titan_builds (stat: Transport endpoint is not connected)
 231     titanrt@tcclab1:~> kill -9 2378
 232
 233     Unfortunately SSHFS is very unstable and hangs regularly, at least the 1.7
 234     version used on the master machine, which was released in 2006.  Another
 235     thing that may help:
 236
 237     titanrt@tcclab1:~> killall -9 sshfs
 238     tcclab1:/home/titanrt/titan_nightly_builds # umount -l vobs/ttcn
 239     tcclab1:/home/titanrt/titan_nightly_builds # umount -l vobs
 240
 241   - It's possible that some kind of network problem prevents the nightly tests to
 242     check out the latest script files. If there're broken links in the test
 243     directory and the test results are missing this might be the case. To make
 244     the script run next morning the freshbuild.sh needs to be run first, since
 245     freshbuild.sh is run by cron.
 246
 247 Licenses
 248 --------
 249
 250 The script can be run by anyone.  Currently, `titanrt' is used for this
 251 purpose, however, the actual user can really be anyone, since it is set from
 252 the environment.  It is a general user available on most of our supported
 253 platforms, it can be created manually in need.  To eliminate user dependence,
 254 all non-absolute paths in the configuration file are prefixed automatically
 255 with the current user's home directory.
 256
 257 To run the tests on all of the supported platforms we're using `hostid' based
 258 license files.  The license file numbers at the moment:
 259
 260   tcclab1 (172.31.21.7, 0x67666473): 4812
 261   tcclab2 (172.31.21.49, 0x67666473): 4812
 262   rhea (159.107.193.33, 0x83dbd963): 5628
 263   bangjohansen (172.31.21.76, 0x380f076e): 2551
 264
 265 Statistics generation
 266 ---------------------
 267
 268 Please note, that the statistics automatically generated by the test system
 269 are useless, unless they're monitored continuously.  Invalid failures will not
 270 be taken into account by the test system.  The statistics are extracted from
 271 the `report.txt' files.  Only the `Reg. tests', `Func. tests', `Perf. tests'
 272 are important.  Weekends are never counted, but holidays need to be handled by
 273 hand.
 274
 275 Rules:
 276
 277 All of them pass  -> 2 points
 278 Any of them fails -> 0 points
 279 Any of them lost  -> 1 points
 280
 281 Sample output:
 282
 283 First period: 2011-03-01 03:01:00
 284 Overall score: 90/100 (90%) Commitment
 285 This period: 2011-06-01 03:01:00
 286 Period score: 9/10 (90%) Commitment