List of files ------------- - titan_builder.sh: Wrapper script for titan_builder.py. It sets some basic environment variables for platforms with tools installed to non-standard locations. - titan_builder.py: Build process control functions. It shouldn't be run directly, use the wrapper script instead. - titan_builder_cfg.py: The configuration file. It contains the build configurations, build slaves etc. - product_handler.py: Helper functions to build the VOB products. - titan_publisher.py: Different publishing related functions. - utils.py: General purpose functions. Brief overview -------------- When titan_builder.sh is started, it checks out a build configuration specific version of TITAN from CVS. TITAN gets configured according to the build configuration (e.g. variables in Makefile.cfg). When it's ready, TITAN is distributed to each slave attached to the given build configuration. The builds are performed sequentially. All script files are distributed as well and the script starts executing in slave mode on the slave machine. The master and the slave can be the same machines. In this case, using different build directories for them is strongly recommended. When all passes are finished on the slave machine the master starts the build on the next slave machine etc., then the remaining build configurations are built. After all that, the master collects all logs from the slave machines. The slaves can produce three kinds of outputs: CSV, TXT, HTML. The CSV file is essential, since the master will consider each slave lost without such a file. The CSV file is used to generate the e-mail notification message as well. Finally, the master publishes the logs and cleans things up. The compiled sources will remain available on each slave until the next build. To run VOB product tests the VOB or a similar directory structure of products should be available on master machine. SSHFS can be used for this purpose. The performance test package is not part of the official TITAN test suite, hence it needs to be distributed for the slaves as well. Currently it's just an archive reachable on the master machine. It's recommended to use configspec 129 and the TCC_Common VOB. All shell commands are executed using the `run_cmd()' function. It has a timeout parameter to kill the command. To make sure, that always the latest version of the scripts runs, one can use a simple script, which checks out the sources from CVS first: #!/bin/bash rm -rf *.{py,sh} TTCNv3/etc/autotest cvs co TTCNv3/etc/autotest && ln -sf TTCNv3/etc/autotest/*.{py,sh} . ./titan_builder.sh -c tests_on_x86_64_linux,tests_on_solaris,tests_on_i386_solaris,limited_tests_on_ubuntu910 1>/dev/null Prerequisites ------------- - Python >= 2.3 - Configured password-less SSH connections between the master and the slave - For scheduled execution use `cron' Command line options -------------------- -c [c1,c2,...]: Select the build configurations to run. If no build configuration is selected, then all configurations will be run. It's a comma separated list. -d: Print all the available build configurations present in the configuration file. Slaves attached to a given build configuration are listed as well. -h: Display available options. -s: Start the script in slave mode. This option should only be used by titan_builder.py, unless everything is in place. Configuration ------------- It's a Python source file, take care of the syntax. - Common options buiddir:STR The build directory of the master. logdir:STR The logs of the master go here. htmldir:STR All HTML files will be published here. vob:STR The VOB products will be copied from here. archive:INT Archive the logs after a specified number of days. cleanup:INT Move the archived logs somewhere else after a specified number of days. measureperiod:INT Reset scores after a given number of days. cleanupslave:{slave:STR,dir:STR} The destination of abandoned logs. - Recipients {name:STR,address:STR} The names and e-mail addresses of the recipients. - Slaves slaves[name:STR]:{ip:STR,user:STR,configs:LIST} The IP address, the user's name, the build configurations attached to the slave. Currently one build configuration is supported for each slave. - Product descriptions products[kind:STR]:{name:STR,action0:BOOL,action1:BOOL,...} `kind' is always a directory under the VOB's root directory (e.g. TestPorts, ProtocolModules). `name' is the product name. Actions can be enabled individually. The predefined actions are the following: - semantic: `compiler -s' - translate: `compiler' - compile: `make' - run: `make run' - Build configurations version:STR Version of TITAN to use. It can be a CVS tag or date. If it's not set the HEAD will be taken. license:STR Location of the license file. gui:BOOL The `GUI' part in Makefile.cfg. jni:BOOL The `JNI' part in Makefile.cfg. debug:BOOL The `DEBUG' part in Makefile.cfg. compilerflags:STR The `COMPILERFLAGS' in Makefile.cfg. ldflags:STR The `LDFLAGS' in Makefile.cfg. gccdir:STR This will affect `CC' and `CXX'. *cc:STR Value of `CC' in synch with the previous option. *cxx:STR Value of `CXX' in synch with the previous option. flex:STR Replace `FLEX'. perl:STR Location of the `perl' interpreter. bison:STR Replace `BISON'. jdkdir:STR Replace `JDKDIR'. qtdir:STR Replace `QTDIR'. xmldir:STR Replace `XMLDIR'. openssldir:STR Replace `OPENSSL_DIR'. regtest:BOOL Run regression tests. perftest:BOOL Run performance tests. The location of the testsuite must be known, since it's not part of CVS. It should be available for the master locally. perftestdir:STR Location of the performance tests. *cpsmin:INT Minimum CPS value for performance tests. *cpsmax:INT Maximum CPS value for performance tests. functest:BOOL Run function tests. vobtest:BOOL Run product tests. *vobtest_logs:BOOL Save logs for product tests. rt2:BOOL Run tests with both run-times. builddir:STR Everything will be done here. It should be different from the master's. logdir:STR Place of the logs. *pdfdir:STR Local directory to copy .pdf files from. If not present no *.pdf files will be there. If it's an empty string the *.pdf files will be faked with empty files. *xsdtests:BOOL Disable regression tests for `xsd2ttcn'. It's very time consuming. measure:BOOL Enable `quality' measurements. (`*' is for optional fields.) Platform specific settings -------------------------- If some essential tools (e.g. GCC) are installed to non-standard locations on a given platform, all of these environmental settings can be placed into a configuration file `${HOME}/.titan_builder'. This file is always sourced first, before starting the actual build. An example file: #!/bin/bash # Basic configuration for `bangjohansen' (172.31.21.76). PATH=${HOME}/apps/bin:/mnt/TTCN/Tools/gcc-3.4.6-sol10/bin:/mnt/TTCN/Tools/binutils-2.17-sol10/bin:/usr/local/bin:/usr/bin:${PATH} LD_LIBRARY_PATH=${HOME}/apps/lib:/mnt/TTCN/Tools/gcc-3.4.6-sol10/lib:/mnt/TTCN/Tools/binutils-2.17-sol10/lib:/usr/local/lib:/usr/lib:${LD_LIBRARY_PATH} TTCN3_LICENSE_FILE=${HOME}/.TTCN3/license_2536.dat CVSROOT=esekits1064.rnd.ki.sw.ericsson.se:/proj/TTCN/cvs_root CVS_SERVER=/usr/local/bin/cvs CVS_RSH=ssh EDITOR=mcedit export PATH LD_LIBRARY_PATH CVSROOT CVS_SERVER EDITOR TTCN3_LICENSE_FILE CVS_RSH When something goes wrong ------------------------- E.g. no e-mail message is received, garbage or missing sections in the e-mail or on the generated HTML page... - Possible reasons and possible solutions - Problems with e-mail system: In case of Postfix check the `mailq' command. All messages still in the queue should be printed. An example output of the `mailq' command: -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient------- 5A57D160B02 1696 Fri Oct 16 09:32:05 ferenc.kovacs@ericsson.com (host mwux020.eth.ericsson.se[159.107.148.18] said: 452 4.4.5 Insufficient disk space; try again later (in reply to MAIL FROM command)) ferenc.kovacs@ericsson.com To clear all the messages: `postsuper -d ALL'. - Exception in the Python code: Check the titan_builder.err-log file of the master and the slaves. All exceptions should go into this file. The other log file titan_builder.log can be useful as well. - Disk quota exceeded. - The SSHFS mounted partitions (e.g. used for HTML publishing) get umounted. The common cause is a segfault: Dec 12 07:00:50 tcclab1 kernel: sshfs[20289]: segfault at 6507a ip 7fbf99315064 sp 44288ea8 error 4 in libc-2.5.so[7fbf992a2000+139000] Check /var/log/messages for details. If SSHFS locks up and the mounted partitions cannot be umounted like: titanrt@tcclab1:~> fusermount -u ~/public_html fusermount: failed to unmount /home/titanrt/public_html: Device or resource busy Simply `killall -9 sshfs' and try to remount the partitions. If SSHFS locks up you may not be able to execute any command in the parent directory of these mount points. The commands will simply hang. If it doesn't work, try the following: titanrt@tcclab1:~> lsof | grep public_html lsof: WARNING: can't stat() fuse file system /home/titanrt/public_html Output information may be incomplete. bash 2378 titanrt cwd unknown /home/titanrt/public_html/titan_builds (stat: Transport endpoint is not connected) ssh 13480 titanrt cwd unknown /home/titanrt/public_html/titan_builds (stat: Transport endpoint is not connected) titanrt@tcclab1:~> kill -9 2378 Unfortunately SSHFS is very unstable and hangs regularly, at least the 1.7 version used on the master machine, which was released in 2006. Another thing that may help: titanrt@tcclab1:~> killall -9 sshfs tcclab1:/home/titanrt/titan_nightly_builds # umount -l vobs/ttcn tcclab1:/home/titanrt/titan_nightly_builds # umount -l vobs - It's possible that some kind of network problem prevents the nightly tests to check out the latest script files. If there're broken links in the test directory and the test results are missing this might be the case. To make the script run next morning the freshbuild.sh needs to be run first, since freshbuild.sh is run by cron. Licenses -------- The script can be run by anyone. Currently, `titanrt' is used for this purpose, however, the actual user can really be anyone, since it is set from the environment. It is a general user available on most of our supported platforms, it can be created manually in need. To eliminate user dependence, all non-absolute paths in the configuration file are prefixed automatically with the current user's home directory. To run the tests on all of the supported platforms we're using `hostid' based license files. The license file numbers at the moment: tcclab1 (172.31.21.7, 0x67666473): 4812 tcclab2 (172.31.21.49, 0x67666473): 4812 rhea (159.107.193.33, 0x83dbd963): 5628 bangjohansen (172.31.21.76, 0x380f076e): 2551 Statistics generation --------------------- Please note, that the statistics automatically generated by the test system are useless, unless they're monitored continuously. Invalid failures will not be taken into account by the test system. The statistics are extracted from the `report.txt' files. Only the `Reg. tests', `Func. tests', `Perf. tests' are important. Weekends are never counted, but holidays need to be handled by hand. Rules: All of them pass -> 2 points Any of them fails -> 0 points Any of them lost -> 1 points Sample output: First period: 2011-03-01 03:01:00 Overall score: 90/100 (90%) Commitment This period: 2011-06-01 03:01:00 Period score: 9/10 (90%) Commitment