Move artifacts from the file system to a database

This topic describes how to configure Deploy to store and retrieve artifacts using two local storage repository formats.

The two local storage repositories are:

• file: The artifacts are stored on and retrieved from the file system.
• db: The artifacts are stored in and retrieved from a relational database management system (RDBMS).

Deploy can only use one local artifact repository at any time. In the deploy-repository.yaml file, you can set the xl.repository.artifacts.type configuration option for the storage repository to either "file" or "db".

xl:
  repository:
    artifacts:
      type: file | db

For more information, see Deploy Properties.

Moving artifacts

When Deploy starts, it checks if any artifacts are stored in a storage format that is not configured. If artifacts are detected, Deploy checks the xl.repository.artifacts.allow-move configuration option to see if the detected artifacts should be moved.

If xl.repository.artifacts.allow-move is set to the default false setting, Deploy does not start and generates an error. Deploy interprets this as an error made in the configuration.

To adjust the configuration: choose another local artifact repository type or set the xl.repository.artifacts.allow-move configuration option to true. After the configuration change, you must restart Deploy.

If xl.repository.artifacts.allow-move is set to true, Deploy starts up and moves the artifacts to the configured local artifact repository. Deploy uses both the configured local artifact repository (example: db) and the not-configured local artifact repository (example file) to retrieve artifacts during the move process. This makes the artifacts that are waiting to be moved available to the system.

The artifact migration process moves the data in small batches with pauses between every two batches. This enables the system to be used for normal tasks during the process.

Handling errors and process restart

If an artifact cannot be moved because an error occurs, a report is written in the log file and the process continues. When Deploy is restarted during the process of moving the artifacts, the startup sequence described earlier will be re-executed. If the xl.repository.artifacts.allow-move option is set to true, the move process will start again. Any artifacts that failed during the previous run, will be re-processed.

When the move process has completed successfully and all artifacts have been moved, a report is written in the log file and the xl.repository.artifacts.allow-move option can be set (or reset) to false. When artifacts are moved from the file system, empty folders may remain in the configured xl.repository.artifacts.root. These empty folders have no impact and you can manually delete them.

Files can remain on the file system, but are not detected as artifacts. This happens when files are no longer in use by the system, but have not been removed. For example, files from application versions that are no longer used. You can remove the files after creating a backup.

If you are upgrading from a version that is earlier than Deploy 8.0.0, restart the server again after migration has finished to ensure that the artifacts are moved. Once the server has started you should see the following in your logs:

        2018-08-17 15:19:54.323 [xl-scheduler-system-pekko.actor.default-dispatcher-2]
        {sourceThread=xl-scheduler-system-pekko.actor.default-dispatcher-4, pekkoSource=pekko://xl-scheduler-system/user/ArtifactsMover, sourceActorSystem=xl-scheduler-system, pekkoTimestamp=13:19:54.320UTC}
        INFO  c.x.d.r.s.a.m.ArtifactsMoverSupervisor - Found artifacts to move: 25 artifacts from file to db.

And a series of the following:

        2018-08-17 15:19:54.588 [xl-scheduler-system-pekko.actor.default-dispatcher-2] {} INFO  c.x.d.r.s.a.m.FileToDbArtifactMover - Moved file artifact. [1 + 0 failed /25]
        2018-08-17 15:19:54.716 [xl-scheduler-system-pekko.actor.default-dispatcher-2] {} INFO  c.x.d.r.s.a.m.FileToDbArtifactMover - Moved file artifact. [2 + 0 failed /25]

If you enable xl.repository.artifacts.allow-move but you do not see the above logs, restart the server. If after restarting the server you still do not see the above logs, contact support.