Migrate Archived Tasks to SQL Database

As of Deploy version 8.0.0, Deploy does not use JCR as the underlying datastore technology. Any upgrades from a pre-8.0.0 Deploy installation require a separate migration procedure, outlined here. As part of this migration process, the archived tasks that Deploy shows reports on, are moved to an SQL database.

This document describes the migration procedure for archived tasks.

Using separate databases

All reports in 8.0 and later are based on data from the SQL reporting database, which may be configured to be different than the one Deploy uses for its live data. The purpose of this separation is that the size of the live database may be reduced, achieving improved performance, faster recovery after a crash, more efficient reporting, and smaller load on the live database.

important

If you want to have a separate reporting database, make sure you set up the database configuration correctly before starting the migration. By default, Deploy will reuse its live database for archived tasks.

Migration process

Except from configuring the database connection, the archive migration is a fully transparent operation running as a background process that takes place during normal Deploy operation, as part of the main migration process.

The Deploy data cannot be moved all at once. During the migration period, the reports on past tasks may be incomplete. Data is migrated from newest to oldest and the reports on recent data will be available first.

The migration process starts automatically when you launch Deploy. The system remains available to use during the migration with a possible small impact on performance. It is recommended to perform the migration during a low activity period (example: night, weekend, or holiday).

Depending on the size of the data you want to migrate, the process can take from minutes to a few days to fully complete. Example: approximately 6000 records in 45 minutes and approximately 180.000 records in 20 hours. The duration of the entire process depends on the sizing of the machine or environment, the usage of the system, etc.

Notes:

1. During migration some messages will be shown in the log.
2. Tasks that cannot be migrated, will be exported to XML files.
3. If Deploy is stopped during migration, the process continues after restart.

Migration process steps

The migration process uses three markers on archived tasks to guide the process: apart from being unmarked, a task can be marked with a migration status of Migrated, Failed to insert, or Failed to delete, according to the process outlined here. The handling of these migration statuses is designed to ensure progress and prevent duplicate work or unnecessary retries.

The migration consists of four consecutive phases:

1. Prepare for insertion: All tasks marked as Failed to insert on a previous migration attempt, will be unmarked. Deploy will retry to migrate these in the second phase. If the migration fails for an archived task due to an external issue, and this issue has now been lifted, this step ensures that another migration attempt will be made on that task.

2. Insert data into the SQL database: Each task not marked as Migrated or Failed to delete during a previous run, is transferred to the SQL database. When successful, the task will be marked as Migrated. When unsuccessful, it will be marked as Failed to insert.

3. Export failed archived tasks to XML: As a fallback, all tasks marked as Failed to insert will be exported as XML.

4. Delete migrated tasks from JCR: Each task marked as Migrated is removed from the JCR repository. If deletion fails, it will be marked as Failed to delete.

Handling of each phase

Each phase is performed in batches that are allowed to run for a specified amount of time. By default, a new batch is triggered every 15 minutes and allowed to run for 9 minutes. For a better performance of the JCR and SQL databases, each batch is divided in sub-batches of a default size 500 and a pause with the default value of 1 minute is inserted between the sub-batches. Each sub-batch queries the JCR database for archived tasks with a particular migration status (for example: Migrated or Failed to insert).

Operational controls

The migration process can be optimized through JMX, using tools such as JConsole, VisualVM, Java Flight Recorder, or others. Deploy provides an MBean called MigrationSettings under the namespace com.xebialabs.xldeploy.migration.

For each phase, the batching schedule can be set using a valid Cron expression. The timeout for each batch, the sub-batch size, and the inter-sub-batch interval can be modified for each phase separately. The changes take immediate effect, providing you with multiple options to reduce pressure on the JCR and SQL databases on a running Deploy system, or to shorten the total migration time.

There is a JMX operation available that allows you to restart the migration without shutting down and restarting the Deploy server (for example: use this when tablespace has run out and the DBA has now added more).

Settings when upgrading Deploy

In the XL_DEPLOY_SERVER_HOME/centralConfiguration/deploy-repository.yaml and XL_DEPLOY_SERVER_HOME/centralConfiguration/deploy-reporting.yaml files (not included in the installation), add a section with the configuration of the reporting database:

xl:
  reporting:
    db-driver-classname: org.postgresql.Driver
    db-password: "DB_PASSWORD"
    db-url: "jdbc:postgresql://DB_URL"
    db-username: "DB_USER"

If you are using MySQL, a configuration property is required:

xl:
  reporting:
    database:
      db-driver-classname: "com.mysql.jdbc.Driver"
      db-password: "DB_PASSWORD"
      db-url: "jdbc:mysql://DB_URL?useLegacyDatetimeCode=false"
      db-username: "DB_USER"

You might require other configuration properties, depending on your setup. For more information, see Deploy Properties

important

To use Deploy with a supported database, ensure that the JDBC driver JAR file is located in XL_DEPLOY_SERVER_HOME/lib or on the Java classpath. For more information, see Configure the Deploy repository.

For more information about upgrading Deploy, see Upgrade Deploy.

Known issues

In some cases the migration process can report an error during the deletion phase. These errors can be safely ignored:

2017-11-28 21:35:11.716 [xl-scheduler-system-akka.actor.default-dispatcher-18] {sourceThread=scala-execution-context-global-267, akkaTimestamp=20:35:11.709UTC, akkaSource=akka://xl-scheduler-system/user/$a/JCR-to-SQL-migration-job-delete-3/$b, sourceActorSystem=xl-scheduler-system} ERROR c.x.d.m.RepeatedBatchProcessor - Exception while processing archived tasks
com.xebialabs.deployit.jcr.RuntimeRepositoryException: /tasks/.....
        at com.xebialabs.deployit.jcr.JcrTemplate.execute(JcrTemplate.java:48)
        at com.xebialabs.deployit.jcr.JcrTemplate.execute(JcrTemplate.java:26)
       .....
Caused by: javax.jcr.PathNotFoundException: /tasks/.....
        at org.apache.jackrabbit.core.ItemManager.getNode(ItemManager.java:577)
        at org.apache.jackrabbit.core.session.SessionItemOperation$6.perform(SessionItemOperation.java:129)
               .....