Version: TeamForge 24.0

Upgrade TeamForge on New Hardware in a Distributed Multi-host Setup

In this distributed setup, TeamForge services are distributed across multiple servers as illustrated in the following table.

server-01 TeamForge Application Server	server-02 TeamForge Database Server	server-03 Review Board Server	server-04 SCM Server	server-05 Code Search Server	server-06 Baseline Server
ctfcore	ctfcore-database	reviewboard¹	subversion	codesearch	baseline²
mail	ctfcore-datamart		gerrit		baseline-post-install³
etl	gerrit-database				baseline-database
search	binary-database
reviewboard-adapter⁴	reviewboard-database
binary	webr-database
cliserver
webr
service-monitor

Here's a list of dos, don'ts and points to remember when you install or upgrade TeamForge.

One-hop Upgrade Compatibility

Though the TeamForge 24.0 installer supports one-hop upgrade from TeamForge 22.1 or later versions, TeamForge 24.0 upgrade instructions, in general, are for upgrading from TeamForge 23.1 (including update releases, if any) to TeamForge 24.0 .

There is no support for one-hop upgrade from TeamForge 22.0 or earlier to TeamForge 24.0 . You must upgrade your site to TeamForge 22.1 or later and then upgrade to TeamForge 24.0 .

Example Multi-hop Upgrade Path

For instance, if you are running TeamForge 18.3, you must follow a multi-hop upgrade path considering the following key dependencies:

TeamForge Upgrade Path: TeamForge 18.3 (PostgreSQL 9.6) → TeamForge 20.3 (PostgreSQL 11.6, JDK 9) → TeamForge 22.1 (PostgreSQL 13.4, Java 11) → TeamForge 24.1 (PostgreSQL 13.15, Java 17)

Source Code Management (Git/Gerrit) Upgrade Path: TeamForge 19.3 (Gerrit 2.15.18) → TeamForge 20.2 (Gerrit 2.16.26) → TeamForge 21.1 (Gerrit 3.2.14) → TeamForge 23.1 (Gerrit 3.5.6)

Operating System Considerations:

TeamForge 23.0 and earlier support RHEL 7.9
TeamForge 23.1 supports RHEL 8.8
TeamForge 24.1 supports RHEL/AlmaLinux 8.10

Before You Begin—Generate New TeamForge License

TeamForge has a new licensing framework starting from TeamForge 21.1. If you are upgrading from TeamForge 21.0 or earlier to TeamForge 21.1 or later, you must get the new TeamForge license and add it to your site before upgrading to TeamForge 21.1 or later. Contact Digital.ai Support to get the new TeamForge license for your site before you run the teamforge provision command. The teamforge provision command fails otherwise.

Before You Begin—CVS End-of-Life

CVS is no longer supported by TeamForge 20.2 and later. You must migrate your CVS repositories to any of the other supported SCM tool (Git/SVN for example) when you upgrade to TeamForge 20.2 or later.

Undeploy CVS on the TeamForge SCM server that runs CVS. Do this after you stop the TeamForge services while upgrading to TeamForge 20.2 or later versions on the same hardware. Skip this step in case of new hardware upgrades.
```
teamforge undeploy -s cvs
```
Remove cvs from the host:SERVICES token of the site-options.conf file (on all the TeamForge servers), failing which the teamforge provision command aborts with an error.

Before You Begin—EventQ End-of-Life

EventQ as a TeamForge service is no longer supported and is completely removed from TeamForge 20.0 (and later). There are a few things to consider in case you have been using EventQ and are upgrading to TeamForge 20.0 or later. For more information, see EventQ End of Life.

Before You Begin—chmod /svnroot

Do this before you stop TeamForge while upgrading to TeamForge 18.2 or later versions.

Get value of SUBVERSION_REPOSITORY_BASE from the /opt/collabnet/teamforge/runtime/conf/runtime-options.conf file of your existing TeamForge server and run the following command:

chmod -R 775 $SUBVERSION_REPOSITORY_BASE

Where $SUBVERSION_REPOSITORY_BASE is the path to the /svnroot directory.

This is required to work around the unusually long time taken to migrate the Subversion data during the first run of the teamforge provision command.

Prepare the New Servers for TeamForge Installation (server-01 through server-07)

Install RHEL 8.10 or AlmaLinux 8.10 and log on as root.
The host must be registered with the Red Hat Network if you are using Red Hat Enterprise Linux.
See the RHEL 8.10 Installation Guide for help.
Check your networking setup. See Set up Networking for more information.

Install the TeamForge Services

Install TeamForge application services on the TeamForge Application Server (server-01).
```
yum install teamforge
```

Install Monit.

important

If you haven't already installed the latest version of the Monit application, download it here.

Download Monit for

RHEL 8.x from the EPEL repository.

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
rpm -ivh epel-release-latest-8.noarch.rpm

Install Monit.
```
yum install monit
```
Install the baseline packages on the TeamForge Application Server (server-01), if you are installing TeamForge Baseline.
```
yum install teamforge-baseline
```
Install the database and baseline packages on the TeamForge Database Server (server-02).
```
yum install teamforge-baseline
```
important

The yum install teamforge-baseline command installs both the database and baseline packages. In case you don't install the TeamForge Baseline, you must use the yum install teamforge-database command to install the database packages." %}

Install Review Board services on the Review Board Server (server-03).
```
yum install teamforge
```
Install SCM services on the SCM Server (server-04).
```
yum install teamforge-scm teamforge-git
```
Install the Code Search service on the Code Search Server (server-05).
```
yum install teamforge-codesearch
```
Install the Baseline packages on the Baseline Server (server-06).
```
yum install teamforge-baseline
```

Back up and Restore TeamForge Database, Data Directories and site-options.conf

See Back up and Restore TeamForge Database, Data Directories and site-options.conf.

Back up and Restore Review Board Database and Data Directories

See Back up and Restore Review Board.

Install PostgreSQL 11 Dependencies

You must install the PostgreSQL 11 dependencies before running the disable object identifiers (OID) script:

Remove the old mount directory from the following location in case it is already mounted.
```
umount /path/to/mount/directory
```
Install the database packages on the TeamForge Database Server (server-02).
```
yum install teamforge-database
```
Provision services.
```
teamforge provision
```
TeamForge 24.0 installer expects the system locale to be LANG=en_US.UTF-8. TeamForge create runtime (teamforge provision) fails otherwise.
Stop TeamForge application ensuring that the PostgreSQL 13 is not up and running.
```
teamforge stop
```
- Run the following command to check if the PostgreSQL postmaster process is still running.
```
ps -ef | grep postmaster
```
Move the existing var data directory to a new location var.new and save it for later use in the provisioning process.
```
mv /opt/collabnet/teamforge/var /opt/collabnet/teamforge/var.new
```
Copy the old var directory to its original location (mount the directory) /opt/collabnet/teamforge/var.
Run the following command to copy data files from the old PostgreSQL 13 var/pgsql data directory to the new PostgreSQL 11 var.new/pgsql data directory.
```
cp -pr var.new/pgsql/* var/pgsql
```
Install the PostgreSQL 11 RPMs that enable you to execute the disable OID script.
```
yum install postgresql11-* -y
```
Start the PostgreSQL 11 instance.
```
pgsql 11 start
```
Provision TeamForge services to configure TeamForge with the new PostgreSQL 11 database.
```
teamforge provision
```

Disable OID While Upgrading to TeamForge 22.0 or later

Do this on the Database Server (server-02) if you are upgrading from TeamForge 21.2 or earlier to TeamForge 22.0 or later.

TeamForge 22.0 supports PostgreSQL 13.15 . As a result, you must run the /opt/collabnet/teamforge/dist/scripts/disable_oid_pg_upgrade13.py script to disable the object identifiers (OIDs) before provisioning TeamForge services.

/opt/collabnet/teamforge/dist/scripts/disable_oid_pg_upgrade13.py

Import the SSL Certs to the Java Keystore

Do this on the TeamForge Application Server (server-01).

Locate the Java keystore.

This is PATH_TO_JAVA/jre/lib/security/cacerts. For example, this may be /usr/local/j2sdk1.4.2_10/jre/lib/security/cacerts.
Locate the Java keytool utility.

This is PATH_TO_JAVA/bin/keytool For example, /usr/local/j2sdk1.4.2_10/bin/keytool.

Import the certificate into the keystore.

PATH_TO_JAVA/bin/keytool -import -keystore PATH_TO_JAVA/jre/lib/security/cacerts -file <server>.crt -alias <server>

note

Any value is accepted for server in -alias <server>.

Set up the site-options.conf File

Log on to the new TeamForge Database Server (server-02) and copy the backed up site-options.conf file to the TeamForge installer directory.
```
cp /tmp/site-options.conf /opt/collabnet/teamforge/etc/
```

Do this on the TeamForge Database Server (server-02).

vi /opt/collabnet/teamforge/etc/site-options.conf

host:SERVICES Token

server-01:SERVICES=ctfcore service-monitor search mail etl binary reviewboard-adapter cliserver webr
server-02:SERVICES=ctfcore-database ctfcore-datamart gerrit-database binary-database reviewboard-database webr-database
server-03:SERVICES=reviewboard
server-04:SERVICES=subversion gerrit
server-05:SERVICES=codesearch
server-06:SERVICES=baseline baseline-post-install baseline-database

tip

Remove server-06 in case you are not installing TeamForge Baseline.

host:PUBLIC_FQDN Token

server-01:PUBLIC_FQDN=my.app.domain.com

Save the site-options.conf file.

For further customization of your site configuration (SSL settings, password policy settings, PostgreSQL settings, LDAP settings and so on):

Provision services.
```
teamforge provision
```
TeamForge 24.0 installer expects the system locale to be LANG=en_US.UTF-8. TeamForge create runtime (teamforge provision) fails otherwise.
Copy the /opt/collabnet/teamforge/etc/site-options.conf file from the TeamForge Database Server (server-02) to the /opt/collabnet/teamforge/etc/ directory of all other servers.

important
Copy the SSL certificate file, SSL chain file, and the TeamForge site's private RSA key file from the TeamForge Application Server (from the path as specified in the site-options.conf tokens SSL_CERT_FILE, SSL_CHAIN_FILE, and SSL_KEY_FILE to Subversion, Gerrit, and Baseline servers.

Provision Services on All the Servers

TeamForge 16.10 and earlier versions use Oracle JDK. As TeamForge 19.2 and later use OpenJDK, the TeamForge installer checks if Oracle JDK is present when you upgrade to TeamForge 19.2 or later—and if found—would error out when you provision TeamForge. You must uninstall Oracle JDK and proceed.

Run the following command to uninstall Oracle JDK:

rpm -e jdk1.8.0_74-1.8.0_74-fcs.x86_64

Provision services.
```
teamforge provision
```
TeamForge 24.0 installer expects the system locale to be LANG=en_US.UTF-8. TeamForge create runtime (teamforge provision) fails otherwise.

You must provision services in a particluar sequence. Usually you start with the Database Server, followed by the Application Server and then by other servers such as SCM, Review Board and Code Search servers.
The TeamForge installer derives this sequence from your site-options.conf file and shows you the order of provisioning servers when you try to provision one of the distributed servers. Follow the exact sequence as instructed.

Provisioning Sequence without Baseline

Provision the Application Server (server-01)
Provision the SCM server (server-04)
Provision the Review Board Server (server-03)
Provision the Code Search Server (server-05)

Provisioning Sequence with Baseline

Provision the Application Server (server-01)
Provision the Baseline Server (server-06)
Copy the /opt/collabnet/teamforge/etc/site-options.conf file from the TeamForge Baseline Server (server-07) to the /opt/collabnet/teamforge/etc/ directory of all other servers.
Provision the Database Server (server-02) again
Provision the Application Server (server-01) again
Provision the SCM server (server-04)
Provision the Review Board Server (server-03)
Provision the Code Search Server (server-05)

Delete Existing Elastic Search Indexes While Upgrading to TeamForge 22.0 or later

TeamForge 22.0 uses Elastic Search 7.16.3 to address the Log4j dependent vulnerabilities. As a result, you must delete the existing Elastic Search indexes as they might not be compatible with Elastic Search 7.16.3. The Elastic Search service would fail in this case. Use the /opt/collabnet/teamforge/runtime/scripts/delete_es_nodes.py script to delete the existing indexes. You must restart the Elastic Search service after deleting the old indexes. For more information, see TeamForge upgrade instructions.

While the existing ELASTICSEARCH_JAVA_OPTS token is still supported to configure the JAVA_OPTS values, the following tokens have been added to configure the minimum and maximum heap size for Elastic Search.

ELASTICSEARCH_MIN_HEAP_SIZE=-Xms2g
ELASTICSEARCH_MAX_HEAP_SIZE=-Xmx2g

In other words, in TeamForge 21.2 and earlier, you just had to configure ELASTICSEARCH_JAVA_OPTS=-Xms2g -Xmx2g -Dlog4j2.formatMsgNoLookups=true.

Whereas, in TeamForge 22.0 and later, you must configure:

ELASTICSEARCH_MIN_HEAP_SIZE=-Xms2g
ELASTICSEARCH_MAX_HEAP_SIZE=-Xmx2g
ELASTICSEARCH_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true

Reinitialize TeamForge

Reinitialize TeamForge on the Review Board Server.
```
teamforge reinitialize
```
During teamforge provision, the Register SCM integration process fails on sites that use self-signed certificates. Perform these steps in such cases.
1. Restart JBoss on the TeamForge Application Server.
```
teamforge restart -s jboss
```
2. Reinitialize TeamForge on the SCM Server.
```
teamforge reinitialize
```
Do you have Git and other SCM tools (SVN) on two separate servers?
Git and other SCM tools (SVN) are typically installed on a server dedicated for SCM. However, if you have Git and SCM (SVN) installed on two separate servers, restart Jboss on the TeamForge Application Server and reinitialize TeamForge on the SCM Server (SVN server) as discussed earlier. In addition, you must also restart TeamForge on the Git Server.

Restart TeamForge on the Git Server: teamforge restart

Finishing Tasks

Verify TeamForge upgrade.
1. Reboot the server and make sure all services come up automatically at startup.
2. Log on to the TeamForge web application using the default Admin credentials.
  - Username: admin
  - Password: admin
3. If your site has custom branding, verify that your branding changes still work as intended. See Customize TeamForge.
4. Let your site's users know they've been upgraded. See Create a Site-wide Broadcast.

Post Upgrade Tasks

Run TeamForge in SELinux enabled Mode

Run the update_artifact_textflex_carriage_return.py Script

Also See...

TeamForge VAR::identifiers.teamforgeVAR::identifiers supports Review Board 4.0.6 on RHEL 8.10. ↩
It's highly recommended that you install the TeamForge Baseline services on a separate server as the baselining process can consume considerable CPU and database resources. ↩
Synchronizes the user information between the baseline database and TeamForge database. ↩
reviewboard-adapter must always be installed on the TeamForge Application Server. ↩

One-hop Upgrade Compatibility​

Example Multi-hop Upgrade Path​

Prepare the New Servers for TeamForge Installation (server-01 through server-07)​

Install the TeamForge Services​

Back up and Restore TeamForge Database, Data Directories and site-options.conf​

Back up and Restore Review Board Database and Data Directories​

Install PostgreSQL 11 Dependencies​

Disable OID While Upgrading to TeamForge 22.0 or later​

Import the SSL Certs to the Java Keystore​

Set up the site-options.conf File​

host:SERVICES Token​

host:PUBLIC_FQDN Token​

Provision Services on All the Servers​

Provisioning Sequence without Baseline​

Provisioning Sequence with Baseline​

Delete Existing Elastic Search Indexes While Upgrading to TeamForge 22.0 or later​

Reinitialize TeamForge​

Finishing Tasks​

Post Upgrade Tasks​

Also See...​

Footnotes​