How to run the Datamart ETL/Dataloader?

This article is for local installs only. It does NOT apply to hosted/SaaS instance. If you have an issue with data in a hosted/SaaS instance, then please contact Support right away!

The Data Mart dataloader, or ETL, is the part of Analytics that reads the Digital.ai Agility database and updates the Data Mart so that you have current data in your reporting database. Some things that are updated when the dataloader is run:

• Users
• user permissions and roles
• new projects added
• projects updated
• themes, features, backlog groups, backlog goals
• custom fields

So with that said, it is important that the dataloader/ETL is run at least daily on production systems or the data will quickly fall behind, and then you are looking at reports that are no longer relevant.

To run the dataloader/ETL, go to the Data Mart folder on the Data Mart server. The folder will be named after your Data Mart instance like this:

[Digital.ai Agility instance name] to [Digital.ai Agility instance name]-datamart

So if your instance name is VersionOneProd you would have this folder: VersionOneProd to VersionOneProd -datamart

Go to that folder and look for VersionOne.DatamartLoader.exe

One thing to note is that there is ALSO a file in there named VersionOne.DatamartLoader.exe.config. If your file manager is hiding the file extensions, then you would see that config file as "VersionOne.DatamartLoader.exe" and think that is the one to run. The only way to tell is that you would have a file right there above it named "VersionOne.DatamartLoader". See? The file extension is hidden. To get your file manager to show file extensions, use a search engine such as Google to find out. Just Google "Show file extensions".

Now, once you have the file, you can do 1 of 3 things.

1. You can double-click that file right there and it will run.
2. You can open a command prompt at that location, and then type in the name and let it run.
3. You can set up Windows Task Scheduler to run it automatically every day.

Obviously we recommend using Option #3.

Also, it is important to note that the dataloader will basically know which mode to run, either Incremental or Full. If you need to force either mode then you can run the dataloader from the command line like this:

VersionOne.DatamartLoader.exe /mode:Incremental

or

VersionOne.DatamartLoader.exe /mode:Full

About Windows Task Scheduler

However, you should NOT tell the task Scheduler to run that EXE directly. This is a known bugaboo with Windows task scheduler. What you SHOULD do is use a batch file that calls the executable. So you have task scheduler called that batch file which calls the executable. That way the dataloader/etl is run as its own process and not as a child process of the task scheduler, which has been known to cause problems with our dataloader. The batch file will look like this:

 

@echo off
rem ***************************************************************************
rem
rem batch file to launch the Digital.ai Agility ETL
rem
rem ***************************************************************************

setlocal

rem ***************************************************************************

rem You need to set these two environment variables based on the ETL location

rem ***************************************************************************

set ETL_DRIVE=C:

set ETL_HOME="\Program Files\VersionOne\VersionOne to VersionOne-datamart"

rem ***************************************************************************

rem this is where all the action happens :)

rem ***************************************************************************

%ETL_DRIVE%

cd %ETL_HOME%

VersionOne.DatamartLoader.exe

endlocal

Please copy the above text in order to create the batch file needed to run the dataload process.

After that, the only thing you need to edit is the ETL_HOME folder location. You would use the location of the VersionOne.DatamartLoader.exe file.

Then set up the task scheduler to call the batch file name.

To monitor the dataloader activity and make sure it is running, you can look at the dataloader log file that will be in that same folder location. The file is called dataloader.log. Each run will append to that file, so you can scroll to the bottom of that file to see the last run. If you see anything other than a message that looks like this:

2016-07-28 06:04:12,916 [1] INFO - Total time: 00:53:15.0410484

Then please log a support ticket and send us that file.