I wrote a blog post Ongoing issues with Surveys (FEB) and IBM Connections which detailed some problems a customer was having with Surveys. This dragged on and resulted in a couple of PMR’s being raised with IBM but I am hopefully at the end of it now.
Recently IBM provided me with a modified .jar to provide additional output when the problem occurred. I needed to add to the ear file. I did this as follows
# cd /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/
# ./wsadmin.sh -lang jython
wsadmin>AdminApp.export(‘Forms Experience Builder’, ‘/tmp/Forms Experience Builder.ear’)
# cp /tmp/Forms\ Experience\ Builder.ear /tmp/Forms\ Experience\ Builder.ear.orig
# mkdir /tmp/feb_expanded
# mkdir /tmp/feb_collapsed
# /opt/IBM/WebSphere/AppServer/bin/EARExpander.sh -ear /tmp/Forms\ Experience\ Builder.ear -operationDir /tmp/feb_expanded/ -operation expand
ADMA4006I: Expanding enterprise archive (EAR) file /tmp/Forms Experience Builder.ear to directory /tmp/feb_expanded/.
# mkdir /tmp/feb_backup
# mv /tmp/feb_expanded/builder.war/WEB-INF/lib/ibm.fsp.core.service.startup-8.0.1.35.jar/ /tmp/feb_backup/
# cp -R /home/ldap/BenW/17891.033.866.ibm.fsp.core.service.startup-8.0.1.81/ /tmp/feb_expanded/builder.war/WEB-INF/lib/ibm.fsp.core.service.startup-8.0.1.35.jar
# /opt/IBM/WebSphere/AppServer/bin/EARExpander.sh -ear ‘/tmp/feb_collapsed/Forms Experience Builder.ear’ -operationDir /tmp/feb_expanded/ -operation collapse
ADMA4007I: Collapsing the contents of directory /tmp/feb_expanded/ to enterprise archive (EAR) file /tmp/feb_collapsed/Forms Experience Builder.ear.
Update the current application using the ISC pointing to /tmp/feb_collapsed/Forms Experience Builder.ear and selecting the default values.
What I found in the SystemOut.log after a period of time was a different error which in the UI was not allowing me to create new surveys but I could complete existing ones which was slightly different to what I was seeing when I raised the PMR. The exception was
[7/11/16 10:34:12:359 BST] 00001b1e StandardExcep E com.ibm.form.nitro.platform.StandardExceptionMapper toResponse ac7d3dec-57f7-482f-83e5-9eaf77c82cbb
java.lang.RuntimeException: Error reading from /tmp/ibm.fsp.temp.1466513524000/fspjars, isDirectory = false, exists = false, canRead = false
at com.ibm.form.platform.service.startup.IsolatingClassLoader.getFileList(IsolatingClassLoader.java:1577)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.access$100(IsolatingClassLoader.java:47)…………..
I created /tmp/ibm.fsp.temp.1466513524000/fspjars and some functionality returned but it wasn’t until I restarted the JVM that it started to work properly.
IBM told me that the problem here is that the /tmp/ directory is getting cleared out and removing the aforementioned directory causing a problem for FEB.
After a bit of Googling I found that tmpwatch was clearing out files/directories that haven’t been edited for 10 days. To stop this I added the bold text.
# vi /etc/cron.daily/tmpwatch
#! /bin/sh
flags=-umc
/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
-x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \
-X '/tmp/hsperfdata_*' -X '/tmp/.hdb*lock' -X '/tmp/.sapstartsrv*.log' \
-X '/tmp/ibm.fsp.*' -X '/tmp/pymp-*' 10d /tmp
/usr/sbin/tmpwatch "$flags" 30d /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
if [ -d "$d" ]; then
/usr/sbin/tmpwatch "$flags" -f 30d "$d"
fi
done
After a few weeks the problem hadn’t manifested again and IBM told me that the cause of the initial PMR was the /tmp directory being emptied. I was dubious at first but then found https://developer.ibm.com/answers/questions/219765/periodically-my-feb-server-stops-working-properly.html which describes problems due the /tmp directory being cleaned out.
As other stuff gets written to the /tmp directory which is what WAS will use by default I decided to use the java.io.tmpdir custom property to instruct WAS to use a directory under /opt/ where it won’t be cleaned by tmpwatch.
Fingers crossed this is the end of it.
collaborationben 