TeamForge CLI Server User Guide

The Digital.ai TeamForge CLI Server makes it possible to customize and extend the TeamForge platform with CLI scripts that run on the TeamForge server. This page covers the security model those scripts run under, how to upload them, how to invoke them, and how to write event handlers and scheduled jobs.

For installation, see the CLI Server Installation page.

Introduction to the CLI Server​

The CLI Server lets you add new web UI actions, extended applications, workflow management, and reporting by running simple CLI scripts on the TeamForge server as extensions to the platform.

CLI scripts can be used as linked applications, tied to web UI actions and buttons, used as pre-commit and post-commit event handlers, and scheduled as jobs that run on a periodic basis.

CLI Server Security​

There are a few differences between normal client usage of the CLI and the way the CLI Server is built, which ensure that scripts run in a secure manner. Keep the following security considerations in mind when you create scripts to run on the server.

Server Aliases and Connectivity​

Every time a CLI script runs, the CLI Server creates a new environment including the server configuration. Only one alias is available to work with, called localhost. This is the local TeamForge instance that the CLI Server is installed on. The localhost alias is never saved between scripts, which ensures that usernames and session keys cannot be reused by another user.

CLI Variables and the Runtime Environment​

Each time a CLI script runs, it gets a fresh environment. Even though the CLI Server is essentially a constantly running instance of the CLI, variables are not persistent between scripts. Every script runs as though the CLI had never been run before.

No Saved .ctf_cli Configuration File​

The CLI Server does not create a .ctf_cli configuration file. Any changes to the server alias list setup or to CLI options are reset to their initial state for each script invocation.

A chroot Jail​

All CLI scripts run in a chroot jail. This means that CLI scripts have no access to anything on the server except what is explicitly added to the CLI script directory tree. In this chroot jail, only /tmp is writeable, so that scripts that need to create files can do so.

There is no automatic cleanup of files in that /tmp directory, so script writers should clean up after themselves. It is always a best practice to use the -tmp option to the output command. The CLI always erases temp files created with -tmp when the script execution is complete.

CLI Server User​

The CLI Server runs in its own user process account called cliserver. This further limits what scripts have access to on the system. For example, a CLI script cannot send signals to any of the other running processes on the system.

No Shell Commands​

Because the CLI Server runs scripts in a chroot jail, it has no access to any other system commands. The CLI shell command is disabled. The CLI Server has access only to files uploaded to the cli directory in the branding repository and to a private tmp directory, so the shell command would be of little use.

Script Administration​

Scripts are added to the CLI Server via the Subversion branding repository. Only users who are given commit access to the /cli directory in that repository can add scripts to the server.

Script and User Authentication​

CLI scripts invoked as linked applications, web UI actions, or event handlers are run as the user who performed the action. However, cron jobs always run as the site administrator. For this reason, limit who you give commit access to the CLI scheduling directory. Ideally these users should already have site administrator permissions, and they should review scripts that users ask to have added to the job scheduler.

Linked applications

When a script is called as a linked application, you must have the SSO checkbox enabled, which is a site administrator only feature. This tells TeamForge to pass over the username of the current user and a one time login token. The CLI automatically creates a localhost alias server entry and connects to it based on those values before the script starts. The username and session key are available as variables to the running script, in case it needs to provide session state to the user.

Web UI actions

To create a button in the web interface that calls a CLI script, use the Velocity API Tool to get the current user and a SOAP session key to pass to the CLI Server. The CLI Server then automatically creates the localhost server alias and connects as that user with that session key before invoking the called script.

Event handlers

All event handlers are invoked as the user who invoked the calling action. This is done in much the same way as web UI actions, by passing a username and session key to the CLI Server.

Scheduled jobs

The CLI job scheduler runs CLI scripts as the site administrator. This is because it has no way to create sessions and run as another user on the system without knowing that user's password.

Uploading CLI Scripts to the Server​

All server based CLI scripts must be uploaded via Subversion to the branding repository in the look project of your TeamForge site. A standard directory structure is pre-configured when the CLI Server is installed.

$ svn checkout --username joeuser https://teamforge.mysite.com/svn/repository-internal/branding branding/cli
A    branding/cli
A    branding/cli/cron
A    branding/cli/cron/weekly
A    branding/cli/cron/daily
A    branding/cli/cron/hourly
A    branding/cli/cron/monthly
A    branding/cli/cron/once
A    branding/cli/events
A    branding/cli/events/precommit
A    branding/cli/events/postcommit
A    branding/cli/actions
A    branding/cli/webapps

To add a CLI script to the server, place the script in the appropriate directory and commit the script into the repository. All scripts are then immediately available for use, events are active right away, and any new jobs are added to the scheduler.

The only portions of the CLI Server that enforce a directory structure are the event handler and the job scheduler. Linked applications and web UI actions can use any structure you create, but it is a best practice to use the base structure provided. Separate linked applications into their own directory under the /cli/webapps path, and place UI actions directly in the /cli/actions directory.

Event Handlers and Controlling When Events Are Invoked​

To attach CLI scripts to events in TeamForge, create a directory structure that reflects the event type and operations. This structure gives you fine control over when events are invoked.

/cli/events/(pre|post)commit/(type|folder)/operation/myScript.ctf

The CLI Server event handler invokes every script found in the appropriate directories. Execution of scripts stops if one of the scripts creates an exception.

Pre-commit and post-commit, synchronous and asynchronous handlers

A pre-commit (synchronous) handler is invoked before an operation is allowed to complete. The most common use for this type of handler is data validation, to help ensure that user submitted data meets specific criteria. The object being updated is in a locked state and cannot be modified.

A post-commit (asynchronous) handler is invoked after the data update has completed. These are typically used to carry out extra actions such as extended workflow, artifact auto-assignment, and rolling up data to parent objects.

Object types and folders

The next directory in the structure can be either the object type or the containing folderId of an object. For example, if you only want the event to be invoked for artifacts in a specific tracker, create a directory with the name of the tracker id.

/cli/events/precommit/tracker1305/update/verifyData.ctf

To have an event invoked for all objects of the same type, use the object identifier, such as artifact, tracker, package, or release.

/cli/events/precommit/artifact/update/verifyData.ctf

There is a special all object type that invokes actions on all objects. This is useful for development purposes but is not recommended for production systems.

Operations

The available operations are all, create, update, and delete. As with the type and folder field, use the all operation sparingly. In a development setup where you need to inspect every event type and operation, you could create a script with the following directory structure.

/cli/events/postcommit/all/all/flagEvent.ctf

Invoking CLI Server Scripts​

Linked applications and web UI actions, that is, scripts called by the click of a UI button or by direct URL, are called using a URI that is the same as the files committed to the branding repository.

For example, a script saved to /cli/actions/copyArtifact.ctf is called with this URL.

http://yoursite.com/cli/actions/copyArtifact.ctf

Linked Applications​

When configuring a script to be used as a linked application, you need a site administrator to enable the SSO flag on the linked application. This allows the CLI to connect as the current TeamForge user.

The process is the same when you add project level linked applications and site-wide linked applications. Enter an appropriate name, the fully qualified URL of the CLI script, and enable SSO. For site-wide applications, the Open Link In selector should typically be set to IFrame, but you can change it as needed.

Web UI Actions and Session Key Handoff​

For web UI actions, the script must be sent a username and SOAP session to work with. Using the API Tool in the Velocity templates makes this straightforward.

#set( $sfSession  = $API60.sessionKey )
#set( $sfUsername = $PAGE_INFO.currentUser.username )

<A href="/cli/actions/createTasks.ctf?sfUsername=${sfUsername}&sfSession=${sfSession}">Create Tasks</a>

Compatibility and TeamForge Issues​

The CLI Server connects to TeamForge using the highest available API version of the TeamForge server it is installed on. There are, however, some known issues with session key handoffs to be aware of.

TeamForge 5.4

This version works normally. Use API50 to get a 5.x session key as in the example above.

TeamForge 6.1

TeamForge 6.1 does not provide a 6.x API tool for Velocity templates, so you have to use the 5.x API session key. This also means that scripts called with a session key need to connect to TeamForge 6.1 using the 5.4 API. Linked applications get a login token and can use the 6.x API normally.

For TeamForge 6.1, use API50 to get a 5.x session key and add a connect line to the top of your CLI script to force the CLI to use the 5.4 API.

set -e session cgi param session
connect local as $sfUsername with key $session use 5.4.0.0

TeamForge 6.1.1

By default, session key handoffs are disabled in TeamForge 6.1.1. You can install a hotfix that enables session handoffs. To get the hotfix, request the RPM from support and ask for release number rel4131. With the session handoff hotfix installed, you can use API60 to get a 6.x session key, and scripts connect normally.

TeamForge 6.2

This version works normally. If you upgraded from previous releases, make sure the hotfix applied in 6.1.1 is removed.

Event Handlers​

The CLI event handler system automatically provides the CLI Server with the credentials of the current user, that is, the username and session key. No special handling or script setup is needed.

Scheduled Jobs​

The job scheduler invokes all scripts as the main site administrator user. This allows scheduled jobs to take advantage of additional features of a TeamForge site, such as read-only SQL access to the database, and user and group administration. This is a powerful feature of the CLI Server, so take care to ensure that only users with site administrator accounts are able to create scheduled jobs.

Preparing a Script to Be Used on the Server​

Preparing a CLI script to be used on the server requires simple modifications compared to a script that you might run on your desktop. The main consideration is that the HTTP protocol expects scripts to return an appropriate header that gives the calling browser some context about the content to be returned.

cgi header text/html

For example, given a CLI script that returns an HTML based report, all that is needed is to add a one line header command to the script. From there the script can be used as before.

cgi header text/html

tmpfilter init tracker
tmpfilter set statusClass Open

go tracker1305 items
  pivot category status
  export -html -all

The example above is a simple report that shows a pivot table for the given tracker. It can be called as a linked application or through a button in the web interface.

The first line of the script tells the user's browser that the content being sent back is HTML. Without this line, expect to see an "Internal Server Error", which means content was returned before the header.

Other Ways to Use HTTP Headers​

There are more advanced features that you can take advantage of in the HTTP protocol. For example, the script below is a simple web action intended to be called from a button while viewing a tracker artifact. When clicked, this CLI script creates a copy of the original artifact, makes the new copy a child of the original artifact, and then redirects the user to the page in the web UI for the new child artifact.

go $sfObject copy as Parent

go $Parent.folderId create
  merge from Parent
  commit
  copy as Child
  return

api TrackerApp->createArtifactDependency $Parent.id, $Child.id, Automatic child creator
cgi redirect /sf/go/$Child.id

In this example the cgi redirect happens last, so it is important that the script have no output before that line.

CGI Parameters and Script Variables​

The CLI Server gives scripts a number of default variables to make scripting easier. Any CGI parameter that starts with the letters sf is automatically converted to a CLI variable of the same name. All other CGI parameters can be accessed with the CLI cgi command.

<A href="/cli/actions/createTasks.ctf?sfUsername=${sfUsername}&sfSession=${sfSession}&trackerId=${trackerId}">Create Tasks</a>

The values passed to the URL above can be accessed this way in the CLI script.

set -e trackerId cgi param trackerId

echo $sfUsername
echo $sfSession
echo $trackerId

Persistent HTML Forms​

Persistent form values means that any value passed to a CLI script to be presented in a form is displayed to the user without any extra effort. For example, a simple search tool may provide the user with an input box to enter a search term. When that form is submitted, the resulting page can still show the form with the value the user entered, along with the search results.

There are two ways to create persistent forms, using the cgi command to create form elements, or placing values in a template with cgi param.

cgi header text/html

cgi start_form
cgi b Find User:
cgi textfield sfSearchTerm
cgi submit sfSubmit, Start Search
cgi end_form

if $sfSearchTerm
do
  find user $sfSearchTerm
    export -html -all username,fullName,email
done

In this example, when the cgi textfield command displays the HTML field element, it puts the value of the CGI parameter sfSearchTerm in as the default value.

<input type="text" name="searchTerm" value="[:ctf:cgi param sfSearchTerm:]">

If you use a template with the HTML pre-defined, use the ctf namespace to run the cgi param command and retrieve the specified form element.

If you need to pre-populate the field value, use the following in your script.

cgi param sfSearchTerm, Some Value

Writing Event Handlers​

CLI scripts used as event handlers should be written in the same way as a linked application. Make sure that they always start with a cgi header line, because TeamForge invokes a CLI event script using the HTTP protocol.

cgi header text/plain

A CLI event script can be used as a pre-commit (synchronous) or post-commit (asynchronous) handler. Pre-commit handlers should be limited to data validation type operations, and the object that the event was invoked for is locked, so you cannot read or change it directly. A pre-commit handler can return messages to the user and can cancel the commit.

Post-commit handlers can make changes to objects, including the object the event was called for. There is no opportunity at that stage to return messages to the user. If your script changes the original object, it is possible that the web interface reloads and displays the original object before your change has been made.

Original and Updated Objects​

Every CLI script that is invoked by an event has two predefined objects that show what the object looked like before and after the change made by the user. This allows scripts to detect what changes were made to the object.

The following script is a post-commit handler invoked on a tracker artifact. In this case it reassigns the artifact to the person who submitted it if someone sets the status to Pending.

cgi header text/plain

if $Original.status ne $Updated.status and $Updated.status eq 'Pending'
do
  go $sfObjectId set assignedTo $Original.createdBy
done

Returning Messages and Canceling an Event​

Pre-commit handlers support passing a message back to the user. Simple messages are displayed in a green notice style banner at the top of the page. Error messages are displayed in a red error banner and cancel the operation.

Basic notice message

cgi header text/plain

message info This worked great.

Error messages

cgi header text/plain, 500

message info This failed badly.

The main difference between the two is the 500 on the cgi header line. The 500 technically means "Internal Server Error". To the event handler this means cancel the current TeamForge operation. If your pre-commit script may cancel an operation, save your cgi header line for later in the script, for when you know it is an error rather than allowing the operation to proceed.

if $Original.status ne $Updated.status and $Updated.status eq 'Pending' and $Updated.assignedTo eq 'admin'
do
  cgi header text/plain, 500
  message info Admin is not an allowed user when status is Pending.
else
  cgi header text/plain
  message info You did good kid.
done

Event Handler Variables​

In addition to the Original and Updated objects, the following variables are available to CLI event handler scripts.

Development Tips​

The CLI Server event handler system logs to a file called ctfevents.log in the TeamForge logs/apps directory. This is accessible in the web UI at /sf/sfmain/do/listLogFile/. The ctfevents.log file can also be downloaded using the CLI systemlog command.

You can enable more debugging output on the fly by creating a file in the system /tmp directory called cliHandler.debug.

touch /tmp/cliHandler.debug

When scripts run, their output is saved to the Apache error log. You can add more data to the Apache log using the message error command in a CLI script. The logs are as follows.

It can be helpful to capture samples of what the Original and Updated objects look like for some objects. Create a script like the following and put it in the appropriate event handler directory for the object and operation you want to capture.

cgi header text/plain

output /tmp/original-$sfEventType.csv
go Original
  export -header
  export
  return
output

output /tmp/updated-$sfEventType.csv
go Updated
  export -header
  export
  return
output

These files are in the CLI Server tmp directory, typically located in the following path.

/opt/collabnet/teamforge/var/cliserver/tmp

You can either access the sample files with a shell account on the server, or use another CLI Server script, such as the Web CLI, to download the results. Once you have saved the sample objects, use the CLI loadcsv command to import the data and inspect it.

Managing Scheduled Jobs​

The CLI Server comes with a web UI for managing the job scheduler. To access it, visit the look project on your site. In that project there should be a linked application called CLI Jobs. Click that to view the scheduler window.

The window shows the current status of the job scheduler and the last log message. The Job Queue box shows jobs that have been triggered to start but are waiting for an open thread to run in. A job can also sit in the queue if the same job is already running.

The Running Jobs box lists all jobs actively running. Each has a red X button that stops the job. Click jobs listed in the queue or running jobs box to see the Subversion history for the job, where you can also view the source code of the job.

Links on the left side of the job control window let you browse CLI jobs, view the complete job scheduler log, and view the main system log.

Manually Starting Jobs​

Any job can be manually added to the job queue. All jobs are known by their interval directory and script name, such as hourly/myscript.ctf.

Using the CLI, connect to your TeamForge server as a site administrator user. Then issue the following command, being sure to use the interval/script.ctf format for your script.

ctf> whois admin ids cliserver set command run hourly/myscript.ctf

The CLI Server job scheduler checks the status of the system once a minute, so it may take up to a minute before you see your command enter the run queue.

You can also stop jobs using the same method. Again, it may take up to a minute before the job is stopped.

ctf> whois admin ids cliserver set command stop hourly/myscript.ctf

Job Control-depend and trigger​

Jobs can depend on other jobs as well as trigger other jobs. This gives scripts greater control over how they interact with each other.

Job controls are written into the script using the comments at the start of the script. The first blank line in the script ends the job control section.

# description: This is what this job does.
# depend on: daily/mainscript.ctf
# depend neg: daily/otherscript.ctf
# trigger ok: once/somejob.ctf
# trigger fail: once/failjob.ctf

versioncheck 1.4.2
silent

The depend and trigger lines can each list multiple jobs as a comma separated list.