FAQs on Install/Upgrade/Administration
PostgreSQL deployment fails with the following error during teamforge provision. What should I do?
Error
Deploying postgres...
Create log folders Done 0.00 secs
Create symlink for pgdata folder Done 0.00 secs
Create PostgreSQL control script Done 0.26 secs
Check and Initialise TeamForge Database Your PostgreSQL installation has been found to be outdated and needs to be upgraded to version 11. This can be done automatically.Do you wish to proceed? [y/N]y
Failed
su - postgres -c '/usr/pgsql-11/bin/pg_upgrade -d /opt/collabnet/teamforge/var/pgsql/9.6/data -D /opt/collabnet/teamforge/var/pgsql/11/data -b /usr/pgsql-9.6/bin -B /usr/pgsql-11/bin' failed
Deployment failed. See /opt/collabnet/teamforge/log/runtime/runtime.log for details.
Exception raised:
Cannot bootstrap service 'gerrit-database-performance-postgres' while in status 'not deployed'
Error from the log file
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
The source cluster was not shut down cleanly.
Solution
Run the following commands.
su - postgres -c "/usr/pgsql-9.6/bin/pg_ctl start -D /opt/collabnet/teamforge/var/pgsql/9.6/data/"
su - postgres -c "/usr/pgsql-9.6/bin/pg_ctl stop -D /opt/collabnet/teamforge/var/pgsql/9.6/data/"
teamforge provision
I see the following error, ERROR: relation "django_site" does not exist at character 78, in the postgresql.logs file while installing TeamForge. What is the cause of this error?
This error is cause by an SQL statement in one of the TeamForge scripts that tries to query the Review Board's django_site table even before it gets created. You can safely ignore this error and proceed.
What could cause the teamforge reload command, run immediately after updating the ETL_JAVA_OPTS token, to fail?
Running the teamforge reload command after updating the ETL_JAVA_OPTS token stops the ETL service first in order to deploy, start and initialize the ETL service again. If the ETL service is not stopped within 180 seconds, the teamforge reload command times out and the teamforge reload command cannot recognize the changes to the ETL_JAVA_OPTS token the next time you run the command.
TeamForge upgrade fails when migrating Baseline database to the latest schema. This happens when upgrading from TeamForge 18.3 to 19.0. What should I do?
Error
Run the following SQL command on the Baseline server and then provision TeamForge.
CREATE TABLE snpsht_baseline_field_value
(
baseline_id character varying(32) COLLATE pg_catalog."default" NOT NULL,
id character varying(32) COLLATE pg_catalog."default" NOT NULL,
field_id character varying(32) COLLATE pg_catalog."default" NOT NULL,
is_default_value boolean NOT NULL,
value character varying(255) COLLATE pg_catalog."default",
metastatus_id integer,
display_order integer NOT NULL,
is_deleted boolean NOT NULL,
CONSTRAINT pk_snpsht_baseline_field_value PRIMARY KEY (baseline_id, id)
)
The teamforge apply-permissions command, when run, removes the setcap IDs. What should I do?
The teamforge apply-permissions command, when run, removes the setcap IDs for /var/www-local/fcgi-bin/cliserver.fcgi. As a workaround, run the following command immediately after running the teamforge apply-permissions command:
setcap cap_setuid,cap_setgid,cap_sys_chroot+iep /var/www-local/fcgi-bin/cliserver.fcgi
I have Git and Subversion on separate servers. I am getting a TeamForge system error when I try to access an existing repo. What should I do.
Make sure that TeamForge, Git and Subversion servers have their time and date synchronized.
If you have Git integration on a separate server, both TeamForge and Git servers must have their time and date synchronized. Similarly, If Subversion is on a separate server, both TeamForge and Subversion servers must have their time and date synchronized.
What should I do if the Reports Database migration fails while upgrading to TeamForge with the following error.
Error Message
"Caused by: org.postgresql.util.PSQLException: ERROR: could not create unique index "schema_version_pk"
Detail: Key (surrogate_id)=(1) is duplicated"
- Verify the
schema_versiontable for multiple entries.-
Connect to the Reports Database.
/opt/collabnet/teamforge/runtime/scripts/psql-reporting-wrapper -
Run
select * from schema_version;and verify if theschema_versiontable has two entries such as the following.major minor sp hotfix 17 11 0 0 17 11 0 0 -
If yes, proceed with the following workaround steps.
-
Run the following commands and confirm the
schema_versiontable entries in the file/tmp/datamart_schemaversion.txt/opt/collabnet/teamforge/runtime/scripts/psql-reporting-wrapper
\o /tmp/datamart_schemaversion.txt -
Delete one of the entries.
delete from schema_version where ctid not in (select max(ctid) from schema_version group by major,minor,sp,hotfix); -
Verify that only one entry exists and provision TeamForge.
select * from schema_version;
-
-
Provision services.
teamforge provision
Why am I getting "Could not connect" status for my email and search server?
On the System Tools page, when you see "Could not connect status for search and email servers," you must stop and start your phoenix.sh process.
You may also need to set the JAVA_HOME environment variable to the location of your JDK.
The stop/start Phoenix commands:
sh /opt/collabnet/teamforge/runtime/scripts/phoenix.sh stop
sh /opt/collabnet/teamforge/runtime/scripts/phoenix.sh start
Why are the dynamic images that TeamForge creates broken?
If you have a fresh install of TeamForge and you're noticing that the dynamic images are not correct, you may be missing a library that is needed to create the images.
The easiest way to find this is to check and see if you have the xorg-x11-depreciated-libs rpm installed:
rpm -qva | grep xorg-x11-depreciated
Watch for the results. If you see that you have the xorg-x11-depreciated-libs rpm installed, and after a server restart you're still not seeing the images, please open a support request. If you do not have the xorg-x11-depreciated-libs rpm installed, it can usually be installed by performing a simple up2date xorg-x11-deprecated-libs and restarting TeamForge.
Due to firewall restrictions I cannot send email from James. How can I resolve this?
If James is unable to send email directly due to firewall restrictions, or mail being rejected from the application servers IP address, you may have to configure it to use a gateway mail server to send outgoing messages through.
To do this, you will need to add the following to the <mailet match="All" class="?RemoteDelivery"> directive in the james config file at /opt/collabnet/teamforge/james/james-<version>/apps/james/SAR-INF/config.xml:
<gateway>smtp.example.com</gateway>
<gatewayPort>25</gatewayPort>
You should find these commented out on line 362 of the config file. If your gateway mail server requires authentication to send email, you may also add the following directives:
<username>username</username>
<password>password</password>
Why am I not getting any error messages when executing the Subversion upgrade script?
Error messages may come when Subversion is installed with a dependent package from an unknown source.
The Subversion working copy script assumes that Subversion is installed with the dependent packages from a proper source repository(RHEL/CollabNet). If you install any dependent packages from any unknown source that is not authorized by RHEL/CollabNet, it will result in inconsistency and this cannot be handled by the Subversion working copy script.
Why do I get a JBoss error "failed to start in 240 seconds, giving up now" while installing TeamForge?
You get this error when the system's RAM is less than the minimum recommended value of 4GB. However, it's most likely that JBoss will start within a few minutes.
To make sure that JBoss starts up, check the service.log file using this command:
tail -f /opt/collabnet/teamforge/log/apps/service.log
If you see messages like the following, the TeamForge application will start in a few minutes.
Check Port Available PASSED: Port 4444 on localhost is available
Check Port Available PASSED: Port 4445 on localhost is available
Waiting for application server to start up.. this can take a few minutes.
JBoss crashed with out of memory error, how do I prevent this?
This can indicate that the JVM heap size is set too small.
You can adjust this by changing the -Xms and -Xmx settings of the JBOSS_JAVA_OPTS token in site-options.conf and rebuilding runtime.
This will appear if the JBoss application server has crashed and you find this error in the server.log:
INFO [STDOUT] java.lang.OutOfMemoryError: Java heap space
The default maximum heap size of 640 MB can cause issues on a heavily used site. If the CTF application is the only thing running on the server, you can increase this to half of the total physical ram on the machine. This should still allow enough memory for the OS and other necessary processes. If you are also running the app, database and scm on the same machine a maximum heap size of 1/4 or the total ram maybe a better setting. Determining the right JVM settings for your install will require testing with your particular usage patterns and database.
You can view the current memory usage under the JVM Environment section of the JBoss webconsole at http://<CTF_SERVER>:8080/web-console/. You will need to log in using the CTF admin password.
JBoss status is in 'starting' for a long time. How to have JBoss started successfully?
If JBoss is not started successfully (status remains 'starting' for a long time), you may have to wait until it starts up successfully or you can kill JBoss process (using its PID) and restart it again.
The following error messages show up when you try to start or stop JBoss respectively while its status is still 'starting':
Cannot execute action as another process is holding the lock.
Cannot stop service 'jboss' while in status 'starting'
To kill the JBoss process:
kill 9 <JBoss PID>
Why am I not able to see the status of the Postgres in the collabnet startup script?
You may not be able to see the status of the Postgres if the host name of the HOST_ token is set to localhost in a SaaS multibox setup.
The Teamforge installer fails to add the IP address of the database box to the listen address in the postgresql.conf file if the host name of the HOST_ token is set to localhost in a SaaS multibox setup.
You must add the IP address of the database box to the listen address in the postgresql.conf file.
Why does the SOAP service show "could not connect" on the Server Status page when everything else appears to work?
This can be caused by an incorrect host name in /etc/sourceforge.properties. Rebuilding runtime will correct this, assuming the hostname is set correctly in the site-options.conf file.
This issue can occur when using the restore.py script to restore data from a TeamForge instance with a different hostname.
Why does startup fail or produce errors?
If TeamForge fails to start up, or is starting but is throwing errors on every page, then typically something went wrong during the JBoss bootstrap process.
Fortunately, JBoss logs this process to: /opt/collabnet/teamforge/jboss/jboss-<version>/server/default/log/boot.log.
TeamForge writes its startup and shutdown info, as well as any system errors to /usr/local/ soureforge/log/server.log. If you encounter a system error while using TeamForge, it is logged here. Additionally, if you see an 'exid' string in the application, the Java stack trace for that exid will be logged in this file.
Why do I get a URL "not found" or "moved permanently" error after applying a patch/upgrade?
If you are experiencing a URL "NOT FOUND" or "MOVED PERMANENTLY" error after applying a patch or upgrade, then set Apache ProxyPreserveHost token to ON in the httpd.conf file.
If you have applied a patch or upgrade and are now receiving the following error:
<The document has moved <a href="https://www.<site>/sf/global/jsp/buildtime.html"
format="html" scope="external">here</a>.</p>
<hr> <address>Apache/2.2.3 (Red Hat) Server at www.<site>.com Port 80</address> </body></html>
Not Found
The requested URL /sf/sfmain/do/userPicker/projects.pftool//sfmain/do/listMonitoringUsers/projects.pftool/discussion.
announcements was not found on this server
Or if you are trying to add users to a monitoring list, and are receiving the following error:
Not Found
The requested URL
/sf/sfmain/do/userPicker/projects.pftool//sfmain/do/listMonitoringUsers/projects.pftool/discussion.announcements
was not found on this server.
Set the ProxyPreserveHost token to ON in the httpd.conf file.
How do I require approval for new user accounts?
You can configure the system so that new users can create their own accounts, but the accounts are not activated until a site admin approves them.
To enable this mode of operation, add the following line to /opt/collabnet/teamforge/sourceforge_home/etc/sourceforge_configuration.properties:
sf.approveNewUserAccounts=true
Once this line has been added to the file, restart TeamForge for it to take effect.
Please note that site admins can still create accounts for new users and they will not be held for approval. Also note that the user will receive an email from TeamForge telling them to confirm their password by clicking on the given link, and the link will not work. The password is properly set on account approval.
How does TeamForge use Velocity templates?
Velocity is the templating language that Digital.ai TeamForge uses to render areas of the site with dynamic information.
You can override the instructions contained in any of these Velocity templates by placing a file of the same name in the equivalent path in the branding repository in the look project on your site.
Velocity templates are located in the templates directory in the branding repository.
| Velocity File | Description |
|---|---|
| menu_bar.vm | Controls the rendering of the top bar across all pages in the system. Displays a small login form, the site logo and current user information as well as the search and projects drop down menus. |
| blank_menu_bar.vm | Contains only the top logo, without the menu that appears below it. |
| body_header.vm | Rendered immediately after the opening body tag. If a site requires everything to be contained in some other container, this template can be used. |
| body_footer.vm | Rendered immediately before the closing body tag. If a site requires everything to be contained in some other container, this template can be used. |
| button_bar.vm | Controls the rendering of the bar beneath the menu bar, which contains the 'Quick Jump" link as well as the buttons that appear on any project page (the one containing the applications). Site admin pages, user settings pages (e.g. my workspace, dashboard) and project pages use different sets of buttons that are passed into this template for rendering. |
| content_header.vm | Rendered after the button bar; wraps the actual contents of the page being viewed. |
| content_footer.vm | Rendered before the body footer; wraps the actual contents of the page being viewed. Contains the Copyright notice. |
| sfmain/home.vm | Velocity template that generates the site home page. |
| sfmain/project_home.vm | Velocity template that generates the default project home page. |
What happens when log files get too big?
Log files can grow very large over time. To maintain reasonable log file sizes, log files are rotated automatically on a schedule.
During this automatic log rotation, live logs are archived every day at 00:00.
Archived logs are stored in compressed form in a directory alongside the live log. For example, if live logs are stored at <LOG_DIR>/apps, apache,..., then compressed log archives are stored at <LOG_ARCHIVE_DIR>/apps, apache,....
The directory structure of the log directory is preserved in the log archive directory.
Empty log files are not compressed.
How do I make the monitoring messages be sent from Forge Administrator?
You can change the default behavior for site options by changing the value from "false" to "true" in this statement:
# MONITORING_EMAIL_FROM_ADMINISTRATOR=false
If this site option token is set to true, then "From:" field is the Forge Administrator, else it is from the user who made the change that initialized the monitoring email.
How do I enable post-commit logging?
You do this by editing the post-commit.py file.
Edit the /opt/collabnet/teamforge/runtime/sourceforge_home/integration/post-commit.py file.
Search for log.setLogging(False) and modify the value from False to True.
What is the suggested log configuration for a production system?
To troubleshoot installation issues, the default log4j configuration is set to DEBUG. This can cause the log files to become quite large. Once your system is successfully installed and in use, you should drop the log levels down to INFO.
See Change the Logging Level on Your Site for how to do this.
If you still have a problem with very large log files, you may want to set up log rotation. Log rotation means to move the log files to a compressed archive to keep them under control.
How do I remove the build and test link from TeamForge pages?
The site administrator can remove these links by checking out and editing the branding repository from the look project.
To remove the build and test links from your TeamForge navigation panel, check out the branding repository from look project and reconfigure the links as shown in the following code sample:
You must be logged on as administrator to perform this task.
[branding_stage]$ cd branding
[branding]$ mkdir -p i18n/com/vasoftware/sf/i18n/apps/sfmain
[branding]$ echo "configure_build_and_test.systemUrl.default=" > i18n/com/vasoftware/sf/i18n/apps/sfmain/application.properties
[branding]$ svn add i18n
A i18n
A i18n/com
A i18n/com/vasoftware
A i18n/com/vasoftware/sf
A i18n/com/vasoftware/sf/i18n
A i18n/com/vasoftware/sf/i18n/apps
A i18n/com/vasoftware/sf/i18n/apps/sfmain
A i18n/com/vasoftware/sf/i18n/apps/sfmain/application.properties
[branding]$ svn commit
" Old: "enter paragraph information and use html tags for bullet points.
How do I resolve timeouts when calling web services?
This is due to the requested operation taking longer then your client SOAP stack is configured to wait before throwing a timeout. You will need to reference your client documentation to see how to update the timeout properties of the connection.
For AXIS in java, you can do this via the sun.net.client.defaultReadTimeout property.
System.setProperty("sun.net.client.defaultReadTimeout", "600000"); //10 minute timeout, in ms