Upgrade TeamForge on the Same Hardware with Oracle Database

In this setup, TeamForge, Oracle database and other services are distributed across three servers, server-01 through server-03 as illustrated in the following table.
You can install TeamForge on both RHEL 8.6 and RHEL/CentOS 7.9. In this distributed setup, all the following services are installed on RHEL 8.6 servers.

| server-01 | server-02 |

TeamForge Application Server	Oracle Database Server
ctfcore	ctfcore-database
mail	ctfcore-datamart
etl
search
codesearch
gerrit
gerrit-database
subversion
reviewboard1
reviewboard-database
reviewboard-adapter2
binary
binary-database
cliserver
service-monitor

important

TeamForge Baselines feature is not supported in TeamForge setup with Oracle database.

Here's a list of dos, don'ts and points to remember when you install or upgrade TeamForge.

One-hop Upgrade Compatibility

Though the TeamForge 22.1 installer supports one-hop upgrade from TeamForge 21.1 or later versions, TeamForge 22.1 upgrade instructions, in general, are for upgrading from TeamForge 22.0 (including update releases, if any) to TeamForge 22.1 .

There is no support for one-hop upgrade from TeamForge 21.0 or earlier to TeamForge 22.1 . You must upgrade your site to TeamForge 21.1 or later and then upgrade to TeamForge 22.1 .

Example Multi-hop Upgrade Path

For instance, if you are running TeamForge 18.3, you must follow a multi-hop upgrade path considering the following key dependencies:

TeamForge Upgrade Path: TeamForge 18.3 (PostgreSQL 9.6) → TeamForge 20.3 (PostgreSQL 11.6, JDK 9) → TeamForge 22.1 (PostgreSQL 13.4, Java 11) → TeamForge 24.1 (PostgreSQL 13.15, Java 17)

Source Code Management (Git/Gerrit) Upgrade Path: TeamForge 19.3 (Gerrit 2.15.18) → TeamForge 20.2 (Gerrit 2.16.26) → TeamForge 21.1 (Gerrit 3.2.14) → TeamForge 23.1 (Gerrit 3.5.6) → TeamForge 24.0 (Gerrit 3.6.8) → TeamForge 24.1 (Gerrit 3.7.9) → TeamForge 25.0 (Gerrit 3.9.11)

Operating System Considerations:

• TeamForge 23.0 and earlier support RHEL 7.9
• TeamForge 23.1 supports RHEL 8.8
• TeamForge 24.1 supports RHEL/AlmaLinux 8.10

Before You Begin—Generate New TeamForge License

TeamForge has a new licensing framework starting from TeamForge 21.1. If you are upgrading from TeamForge 21.0 or earlier to TeamForge 21.1 or later, you must get the new TeamForge license and add it to your site before upgrading to TeamForge 21.1 or later. Contact Digital.ai Support to get the new TeamForge license for your site before you run the teamforge provision command. The teamforge provision command fails otherwise.

Before You Begin—CVS End-of-Life

CVS is no longer supported by TeamForge 20.2 and later. You must migrate your CVS repositories to any of the other supported SCM tool (Git/SVN for example) when you upgrade to TeamForge 20.2 or later.

1. Undeploy CVS on the TeamForge SCM server that runs CVS. Do this after you stop the TeamForge services while upgrading to TeamForge 20.2 or later versions on the same hardware. Skip this step in case of new hardware upgrades.

   teamforge undeploy -s cvs

2. Remove cvs from the host:SERVICES token of the site-options.conf file (on all the TeamForge servers), failing which the teamforge provision command aborts with an error.

Before You Begin—EventQ End-of-Life

EventQ as a TeamForge service is no longer supported and is completely removed from TeamForge 20.0 (and later). There are a few things to consider in case you have been using EventQ and are upgrading to TeamForge 20.0 or later. For more information, see EventQ End of Life.

Before You Begin—chmod /svnroot

Do this before you stop TeamForge while upgrading to TeamForge 18.2 or later versions.

Get value of SUBVERSION_REPOSITORY_BASE from the /opt/collabnet/teamforge/runtime/conf/runtime-options.conf file of your existing TeamForge server and run the following command:

chmod -R 775 $SUBVERSION_REPOSITORY_BASE

Where $SUBVERSION_REPOSITORY_BASE is the path to the /svnroot directory.

This is required to work around the unusually long time taken to migrate the Subversion data during the first run of the teamforge provision command.

Back up Your Oracle Database

• See Oracle Database Backup and Recovery User's Guide
• See Oracle Database Backup and Recovery FAQ

Uninstall Custom Event Handlers, Hot Fixes and Add-ons

Log on to the TeamForge Application Server.

1. SOAP 50 is no longer supported. Back up all your custom event handlers and remove all the event handler JAR files before starting your TeamForge upgrade process.
   1. Go to My Workspace > Admin.
   2. Click System Tools from the Projects menu.
   3. Click Customizations.
   4. Select the custom event handler and click Delete.
      tip

      Post upgrade, you can add custom event handlers again from the backup while making sure that you don't have SOAP50 (deprecated) library used.

2. Uninstall hotfixes and add-ons, if any, installed on your site.

yum upgrade

1. Stop TeamForge.

   important

   Stop TeamForge on all the servers in a distributed setup.

   teamforge stop

2. Upgrade the operating system packages.

   yum upgrade

note

Run yum upgrade on all the servers.

Configure the TeamForge Installation Repository

Upgrade the TeamForge Services

1. Upgrade the TeamForge and Review Board application services on the TeamForge Application Server (server-01).

   yum install teamforge

   Install Monit.

   important

   If you haven't already installed the latest version of the Monit application, download it here.

   1. Download Monit for

      • RHEL 8.x from the EPEL repository.

        wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
        rpm -ivh epel-release-latest-8.noarch.rpm

      • RHEL/CentOS 7.x from the EPEL repository.

        wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
        rpm -ivh epel-release-latest-7.noarch.rpm

   2. Install Monit.

      yum install monit

Back up the TeamForge Data Directories

On sites running TeamForge 16.7 or earlier versions:

1. Back up the following data directories.

   tip

   In a distributed setup, you must backup specific directories such as /svnroot and /cvsroot from the server that hosts those SCM services.

   note

   CVS is no longer supported by TeamForge 20.2 (and later). Backing up and restoring the /cvsroot is recommended, but optional though.

   Directory	Contents
   /opt/collabnet/teamforge/var	User-created data, such as artifact attachments
   /opt/collabnet/reviewboard	Review Board data
   /svnroot	Subversion source code repositories
   /sf-svnroot	Subversion repository for branding data
   /cvsroot	CVS source code repositories (required only if you have CVS)
   /gitroot	Git source code repositories

   cp -Rpf /svnroot /sf-svnroot /cvsroot /gitroot /opt/collabnet/teamforge/var /opt/collabnet/reviewboard /tmp/backup_dir

2. Back up the /opt/collabnet/gerrit directory if you have Git integration.

   tip

   Do this on the server that hosts the TeamForge-Git integration services.

: ::

mkdir /tmp/backup_dir/gerrit
cp -Rpfv /gitroot /tmp/backup_dir
cp -Rpfv /opt/collabnet/gerrit/ /tmp/backup_dir/gerrit

On sites running TeamForge 16.10 or later versions:

1. Back up the /opt/collabnet/teamforge/var directory.

   tip

   Do this on both the TeamForge Application and Database servers in case you have them running on two separate servers.

   mkdir -p /tmp/backup_dir
   cp -Rpfv /opt/collabnet/teamforge/var /tmp/backup_dir

2. Back up the /opt/collabnet/gerrit directory if you have Git integration.

   tip

   Do this on the server that hosts the TeamForge-Git integration services.

   mkdir /tmp/backup_dir/gerrit
   cp -Rpfv /opt/collabnet/gerrit/ /tmp/backup_dir/gerrit

Back up and Restore Review Board Database and Data Directories

See Back up and Restore Review Board Database and Data Directories

Set up the site-options.conf File and Provision Services

1. Log on to the TeamForge Application Server (server-01), set up the site-options.conf file, and provision the services.

   vi /opt/collabnet/teamforge/etc/site-options.conf

   host:SERVICES Token

   server-01:SERVICES=ctfcore mail etl service-monitor search subversion codesearch cliserver gerrit gerrit-database  binary binary-database reviewboard reviewboard-database reviewboard-adapter cliserver
   server-02:SERVICES=ctfcore-database ctfcore-datamart

   host:PUBLIC_FQDN Token

   server-01:PUBLIC_FQDN=my.app.domain.com

   Configure the Oracle Database Tokens

   Configure the Oracle database name, usernames and passwords as configured on the Oracle Database Server.

   • Database type is oracle (DATABASE_TYPE=oracle)
   • Database service name is the host name of the Oracle Database Server (for example, DATABASE_SERVICE_NAME=cu349.maa.collab.net)
   • Reports database service name is the host name of the server where the datamart is (for example, REPORTS_DATABASE_SERVICE_NAME=cu349.maa.collab.net)

      DATABASE_TYPE=oracle
   
      # Adjust usernames/passwords to match what has been configured on the database server.
      DATABASE_USERNAME=ctfuser
      DATABASE_PASSWORD=ctfpwd
      DATABASE_READ_ONLY_USER=ctfrouser
      DATABASE_READ_ONLY_PASSWORD=ctfropwd
      DATABASE_NAME=orcl
      DATABASE_SERVICE_NAME=
   ​
      # Adjust usernames/passwords to match what has been configured on the database server.
      REPORTS_DATABASE_USERNAME=ctfrptuser
      REPORTS_DATABASE_PASSWORD=ctfrptpwd
      REPORTS_DATABASE_NAME=orcl
      REPORTS_DATABASE_READ_ONLY_USER=ctfrptrouser
      REPORTS_DATABASE_READ_ONLY_PASSWORD=ctfrptropwd
      REPORTS_DATABASE_SERVICE_NAME=

   Save the site-options.conf file.

   For further customization of your site configuration (SSL settings, password policy settings, PostgreSQL settings, LDAP settings and so on):

2. Provision services.

   teamforge provision

   TeamForge 22.1 installer expects the system locale to be LANG=en_US.UTF-8. TeamForge create runtime (teamforge provision) fails otherwise.

Delete Existing Elastic Search Indexes While Upgrading to TeamForge 22.0 or later

TeamForge 22.0 uses Elastic Search 7.16.3 to address the Log4j dependent vulnerabilities. As a result, you must delete the existing Elastic Search indexes as they might not be compatible with Elastic Search 7.16.3. The Elastic Search service would fail in this case. Use the /opt/collabnet/teamforge/runtime/scripts/delete_es_nodes.py script to delete the existing indexes. You must restart the Elastic Search service after deleting the old indexes. For more information, see TeamForge upgrade instructions.

While the existing ELASTICSEARCH_JAVA_OPTS token is still supported to configure the JAVA_OPTS values, the following tokens have been added to configure the minimum and maximum heap size for Elastic Search.

• ELASTICSEARCH_MIN_HEAP_SIZE=-Xms2g
• ELASTICSEARCH_MAX_HEAP_SIZE=-Xmx2g

In other words, in TeamForge 21.2 and earlier, you just had to configure ELASTICSEARCH_JAVA_OPTS=-Xms2g -Xmx2g -Dlog4j2.formatMsgNoLookups=true.

Whereas, in TeamForge 22.0 and later, you must configure:

• ELASTICSEARCH_MIN_HEAP_SIZE=-Xms2g
• ELASTICSEARCH_MAX_HEAP_SIZE=-Xmx2g
• ELASTICSEARCH_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true

Update the Webhook Events and Create New Webhooks

While TeamForge 19.2 (and earlier) offered simple webhooks support, TeamForge 19.3 (and later) offer pre-submit and post-submit webhooks. New webhook events have been added and webhook event names have been updated as well.

If you are upgrading from TeamForge 19.2 (or earlier) to TeamForge 19.3 (or later), you must truncate the webhook and webhook_event tables on the Oracle database server, insert the new webhook events into the webhook_event table and create new webhooks.

1. Note down the list of webhooks you have before truncating the webhooks table. You may take a screenshot of the webhook configuration just in case.

2. Run the following queries to back up the webhook and webhook_event TeamForge tables.

   create table webhookbackup as select * from webhook;
   create table webhookeventsbackup as select * from webhook_event;

3. Truncate the webhook and webhook_event TeamForge tables.

   truncate table webhook;
   truncate table webhook_event;

4. Insert the new TeamForge webhook events into the webhook_event table. Run the following queries one-by-one in order on the Oracle database server.

   insert  into  webhook_event  (surrogate_id,webhook_surrogate_id,event_type,  event_type_name)  select  webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Create','TOPIC'    from  dual  where  Not  exists  (select  1  from  webhook_event  where  event_type='Teamforge.Artifact.Create');
   
   insert  into  webhook_event  (surrogate_id,webhook_surrogate_id,event_type,  event_type_name)    select  webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Update','TOPIC'  from  dual  where  Not  exists  (select  1  from  webhook_event  where  event_type='Teamforge.Artifact.Update');
   
   insert  into  webhook_event  (surrogate_id,webhook_surrogate_id,event_type,  event_type_name) select  webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Move','TOPIC'  from  dual  where  Not  exists  (select  1  from  webhook_event  where  event_type='Teamforge.Artifact.Move');
   
   insert  into  webhook_event  (surrogate_id,webhook_surrogate_id,event_type,  event_type_name) select  webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Clone','TOPIC'    from  dual  where  Not  exists  (select  1  from  webhook_event  where  event_type='Teamforge.Artifact.Clone');
   
   insert  into  webhook_event  (surrogate_id,webhook_surrogate_id,event_type,  event_type_name) select  webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Delete','TOPIC'    from  dual  where  Not  exists  (select  1  from  webhook_event  where  event_type='Teamforge.Artifact.Delete');
   
   insert  into  webhook_event  (surrogate_id,webhook_surrogate_id,event_type,  event_type_name) select  webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Create.Presubmit','SYNC'   from  dual  where  Not  exists  (select  1  from  webhook_event   where      event_type='Teamforge.Artifact.Create.Presubmit');
   
   insert  into  webhook_event  (surrogate_id,webhook_surrogate_id,event_type,  event_type_name) select  webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Update.Presubmit','SYNC'   from  dual  where  Not  exists  (select  1  from  webhook_event  where  event_type='Teamforge.Artifact.Update.Presubmit');
   
   insert  into  webhook_event  (surrogate_id,webhook_surrogate_id,event_type,  event_type_name) select  webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Move.Presubmit','SYNC'  from  dual  where  Not  exists  (select  1  from  webhook_event   where  event_type='Teamforge.Artifact.Move.Presubmit');
   
   insert  into  webhook_event  (surrogate_id,webhook_surrogate_id,event_type,  event_type_name) select  webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Clone.Presubmit','SYNC'  from  dual  where  Not  exists  (select  1  from  webhook_event   where  event_type='Teamforge.Artifact.Clone.Presubmit');
   
   insert  into  webhook_event  (surrogate_id,webhook_surrogate_id,event_type,  event_type_name) select  webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Delete.Presubmit','SYNC'  from  dual  where  Not  exists  (select  1  from  webhook_event  where  event_type='Teamforge.Artifact.Delete.Presubmit');
   

5. Run the following query to verify the events inserted into the webhook_event table.

   select event_type,  event_type_name from webhook_event;

   The output lists the events:

   

Verify TeamForge Upgrade

• Verify TeamForge upgrade.

  1. Reboot the server and make sure all services come up automatically at startup.
  2. Log on to the TeamForge web application using the default Admin credentials.
     • Username: admin
     • Password: admin
  3. If your site has custom branding, verify that your branding changes still work as intended. See Customize TeamForge.
  4. Let your site's users know they've been upgraded. See Create a Site-wide Broadcast.

Post Upgrade Tasks

• Run TeamForge in SELinux enabled Mode

• Users are not getting email notifications for review requests and reviews. What should I do?
• Integrate Jenkins, JIRA, and TestLink using the TeamForge Webhooks-based Event Broker

Also See...

• FAQs on Install / Upgrade / Administration

Footnotes

1. TeamForge 22.1 supports Review Board VAR::identifiers.review_board on RHEL 8.6 and Review Board 2.5.6.1 on RHEL/CentOS 7.9. ↩

2. reviewboard-adapter must always be installed on the TeamForge Application Server. ↩