End to Surveys problems in IBM Connections 5.0?

I wrote a blog post Ongoing issues with Surveys (FEB) and IBM Connections which detailed some problems a customer was having with Surveys. This dragged on and resulted in a couple of PMR’s being raised with IBM but I am hopefully at the end of it now.

Recently IBM provided me with a modified .jar to provide additional output when the problem occurred. I needed to add to the ear file. I did this as follows

# cd /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/
# ./wsadmin.sh -lang jython
wsadmin>AdminApp.export(‘Forms Experience Builder’, ‘/tmp/Forms Experience Builder.ear’)
# cp /tmp/Forms\ Experience\ Builder.ear /tmp/Forms\ Experience\ Builder.ear.orig
# mkdir /tmp/feb_expanded
# mkdir /tmp/feb_collapsed
# /opt/IBM/WebSphere/AppServer/bin/EARExpander.sh -ear /tmp/Forms\ Experience\ Builder.ear -operationDir /tmp/feb_expanded/ -operation expand
ADMA4006I: Expanding enterprise archive (EAR) file /tmp/Forms Experience Builder.ear to directory /tmp/feb_expanded/.
# mkdir /tmp/feb_backup
# mv /tmp/feb_expanded/builder.war/WEB-INF/lib/ibm.fsp.core.service.startup-8.0.1.35.jar/ /tmp/feb_backup/
# cp -R /home/ldap/BenW/17891.033.866.ibm.fsp.core.service.startup-8.0.1.81/ /tmp/feb_expanded/builder.war/WEB-INF/lib/ibm.fsp.core.service.startup-8.0.1.35.jar
# /opt/IBM/WebSphere/AppServer/bin/EARExpander.sh -ear ‘/tmp/feb_collapsed/Forms Experience Builder.ear’ -operationDir /tmp/feb_expanded/ -operation collapse
ADMA4007I: Collapsing the contents of directory /tmp/feb_expanded/ to enterprise archive (EAR) file /tmp/feb_collapsed/Forms Experience Builder.ear.
Update the current application using the ISC pointing to /tmp/feb_collapsed/Forms Experience Builder.ear and selecting the default values.

What I found in the SystemOut.log after a period of time was a different error which in the UI was not allowing me to create new surveys but I could complete existing ones which was slightly different to what I was seeing when I raised the PMR. The exception was

[7/11/16 10:34:12:359 BST] 00001b1e StandardExcep E com.ibm.form.nitro.platform.StandardExceptionMapper toResponse ac7d3dec-57f7-482f-83e5-9eaf77c82cbb
java.lang.RuntimeException: Error reading from /tmp/ibm.fsp.temp.1466513524000/fspjars, isDirectory = false, exists = false, canRead = false
at com.ibm.form.platform.service.startup.IsolatingClassLoader.getFileList(IsolatingClassLoader.java:1577)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.access$100(IsolatingClassLoader.java:47)…………..

I created /tmp/ibm.fsp.temp.1466513524000/fspjars and some functionality returned but it wasn’t until I restarted the JVM that it started to work properly.

IBM told me that the problem here is that the /tmp/ directory is getting cleared out and removing the aforementioned directory causing a problem for FEB.

After a bit of Googling I found that tmpwatch was clearing out files/directories that haven’t been edited for 10 days. To stop this I added the bold text.

# vi /etc/cron.daily/tmpwatch
#! /bin/sh
flags=-umc
/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
        -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \
        -X '/tmp/hsperfdata_*' -X '/tmp/.hdb*lock' -X '/tmp/.sapstartsrv*.log' \
        -X '/tmp/ibm.fsp.*' -X '/tmp/pymp-*' 10d /tmp
/usr/sbin/tmpwatch "$flags" 30d /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
    if [ -d "$d" ]; then
        /usr/sbin/tmpwatch "$flags" -f 30d "$d"
    fi
done

After a few weeks the problem hadn’t manifested again and IBM told me that the cause of the initial PMR was the /tmp directory being emptied. I was dubious at first but then found https://developer.ibm.com/answers/questions/219765/periodically-my-feb-server-stops-working-properly.html which describes problems due the /tmp directory being cleaned out.

As other stuff gets written to the /tmp directory which is what WAS will use by default I decided to use the java.io.tmpdir custom property to instruct WAS to use a directory under /opt/ where it won’t be cleaned by tmpwatch.

Fingers crossed this is the end of it.

IBM Sametime Video Manager start up scripts

I managed to get my hands on a restart script from IBM PMR L3 to start up SolidD and the Video Manager at OS start up and thought that I should share it since it can be a little daunting trying to put together a script on an OS that for some may be quite new to them.

The Video Manager uses SolidDB which needs to be be started first before WAS starts. This involves creating start up scripts, registering them with chkconfig and then changing the start up order.

These scripts are designed for Linux so RHEL (or CentOS). I don’t believe they will work for SUSE Linux Enterprise Server (SLES).

The script for WAS will allow you to stop the application server but it will not allow me to stop SolidDB that needs to be done manually. I’m sure it can be tweaked to work but these are for OS start up and they work for that use case.

standalone_eval_server_start_init.sh

# vi /opt/solidDB/soliddb-7.0/standalone_eval_server_start_init.sh

###################

#!/bin/sh
# *********************************************************************************************************
# ** Description : Shell script to start solidDB evaluation process after machine reboot
# ** Launches solidDB server process with default network listen name: tcp 2315
# ** creates error file boot_error.log in the /opt/solidDB/soliddb-7.0 in case of error
# ** Assumption : 1. Directory /opt/solidDB/soliddb-7.0/eval_kit/standalone is present
# **                    : 2. In Directory /opt/solidDB/soliddb-7.0/eval_kit/standalone ,solid.db file is present
# **********************************************************************************************************
SOLID_DIR=/opt/solidDB/soliddb-7.0
today=`date +”%m-%d-%y”`
boot_error_file=$SOLID_DIR/boot_error.log
err_msg_no_dbfile_exist=”No database files solid.db exists in eval_kit/standalone exists , could not start solid db.”
err_msg_dir_path=”Directory structure is not correct . Please check if eval_kit/standalone are present. could not start solid db.”

# Check if the script is started in the right place
if [ -d $SOLID_DIR/eval_kit/standalone ]; then
# locate the executables directory
cd $SOLID_DIR/bin
binpath=`pwd`
cd ..
rootbytes=`pwd | wc -c`
bindir=`echo $binpath | cut -c $rootbytes- | cut -c 2-`

# check if the database exists already
if [ -f $SOLID_DIR/eval_kit/standalone/solid.db ]; then
$bindir/solid -c eval_kit/standalone &

else # default database file did not exist
echo “$today : $err_msg_no_dbfile_exist” >> “$boot_error_file”
exit 1
fi
else # directory structure is not correct
echo “$today : $err_msg_dir_path” >> “$boot_error_file”
exit 1
fi

# End of script.

###################

# chmod +x /opt/solidDB/soliddb-7.0/standalone_eval_server_start_init.sh

SolidDB.init

# vi /etc/init.d/SolidDB.init

###################

#!/bin/sh
#

# IBM Confidential OCO Source Material

# The next lines are for chkconfig on RedHat systems.
# chkconfig: 2345 97 03
# description: Starts and stops Solid db instance \
#              instances.
# The next lines are for chkconfig on RHEL systems.
### BEGIN INIT INFO
# Provides: standalone_eval_server_start_init.sh
# Required-Start:
# Required-Stop: $STMediaServer_was.init
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Starts and stops Solid db instance
### END INIT INFO

# START BLOCK
SOLID_DIR=”/opt/solidDB/soliddb-7.0″
solid_init=”standalone_eval_server_start_init.sh”
solid_stop=”standalone_eval_server_stop”
log_file=”/opt/solidDB/soliddb-7.0/boot_log”
today=`date +%Y_%m_%d`
# END BLOCK

RETVAL=0

start_solid()
{
echo “$today” >> $log_file
startCmd=”${SOLID_DIR}/${solid_init}”
if [ -f “${startCmd}” -a -x “${startCmd}” ] ; then
echo “Starting Solid db instance …” >> $log_file
“${startCmd}”
else
echo “Failure starting Solid db instance…” >> $log_file
echo “The service definition may be invalid – script ${startCmd}” >> $log_file
echo “could not be found or was not executable.” >> $log_file
fi
}

stop_solid()
{
echo “$today” >> $log_file
stopCmd=”${SOLID_DIR}/${solid_stop}”
if [ -f “${stopCmd}” -a -x “${stopCmd}” ] ; then
echo “Stopping Solid db instance …” >> $log_file
“${stopCmd}”
else
echo “Failure starting Solid db instance…” >> $log_file
echo “The service definition may be invalid – script ${startCmd}” >> $log_file
echo “could not be found or was not executable.” >> $log_file
fi
}

case “$1” in
start)
shift
start_solid
;;

stop)
shift
stop_solid
;;

restart)
stop_solid
start_solid
;;

*)
echo “Usage: $0 {start|stop|restart}”
exit 1
;;
esac

if [ $RETVAL -ne 0 ]; then
echo exit code: $RETVAL >> $log_file
fi

exit $RETVAL

###################

# chmod 755 /etc/init.d/SolidDB.init
# chkconfig –add SolidDB.init
# chkconfig –level 35 SolidDB.init on

# chkconfig –list | grep -i solid
SolidDB.init    0:off   1:off   2:off   3:on    4:off   5:on    6:off

Video Manager

Change WAS_HOME to match your server.

# vi /etc/init.d/VMgr

###################

#!/bin/bash
#
# apache
#
# chkconfig: 5 90 10
# description: Start up the WebSphere Application Server.
RETVAL=$?
WAS_HOME=”/opt/IBM/WebSphere/AppServer/profiles/HOSTSTMSPNProfile1″
# added line to ensure that environment variables are set correctly
. /etc/profile
case “$1″ in
start)
if [ -f $WAS_HOME/bin/startServer.sh ]; then
echo $”Starting IBM WebSphere STMediaServer”
$WAS_HOME/bin/startServer.sh STMediaServer
fi
;;
stop)
if [ -f $WAS_HOME/bin/stopServer.sh ]; then
echo $”Stop IBM WebSphere STMediaServer”
$WAS_HOME/bin/stopServer.sh STMediaServer -username wasadmin -password *************
fi
;;
status)
if [ -f $WAS_HOME/bin/serverStatus.sh ]; then
echo $”Show status of IBM WebSphere STMediaServer”
$WAS_HOME/bin/serverStatus.sh -all -username wasadmin -password ********
fi
;;
*)
echo $”Usage: $0 {start|stop|status}”
exit 1
;;
esac
exit $RETVAL

###################

# chmod 755 /etc/init.d/VMgr
# chkconfig –add VMgr
# chkconfig –level 35 VMgr on

Start up order

The numbers shown after the slash indicate the start up order. The nearer to zero the sooner it starts up. In the following examples S90VMgr starts up before S97SolidDB.init which is not what is wanted. We want SolidDB to start first so by renaming the files we can manipulate the start up order.

# cd /etc/rc.d
# find . -iname “*solid*”
./rc1.d/K03SolidDB.init
./init.d/SolidDB.init
./rc0.d/K03SolidDB.init
./rc4.d/K03SolidDB.init
./rc6.d/K03SolidDB.init
./rc5.d/S97SolidDB.init
./rc3.d/S97SolidDB.init
./rc2.d/K03SolidDB.init

# find . -iname “*VMgr*”
./rc0.d/K10VMgr
./rc2.d/K10VMgr
./rc6.d/K10VMgr
./rc5.d/S90VMgr
./rc1.d/K10VMgr
./rc3.d/S90VMgr
./init.d/VMgr
./rc4.d/K10VMgr

Change start up order

These steps change the start up order so that SolidDB starts before WAS.

# cd /etc/rc.d/rc3.d/
# mv ./S97SolidDB.init ./S90SolidDB.init
# mv ./S90VMgr ./S97VMgr

# cd /etc/rc.d/rc5.d/
# mv ./S97SolidDB.init ./S90SolidDB.init
# mv ./S90VMgr ./S97VMgr