Overview

The Workflow Import Queue is a new, scalable, Workflow-centric background processing queue for importing CSV data into LearningBuilder introduced in 10.4.0.

When compared to the legacy upload features, the the primary benefit of the Workflow Import Queue is that it leverages the Workflow Engine to make many new things possible:

• Validation Rules and Required Fields enforce complex and dynamic validation requirements;

• Uploads can trigger Workflow Actions;

• New types of data can be imported (e.g. Learning Plan Instances);

• Administrators can upload data directly through the Admin area of LearningBuilder, rather than needing to use the Provider area.

The Workflow Import Queue also supports larger files with more predictable performance.

Lastly, a much improved UI provides more transparency into and control over the entire process. Users can have more confidence that their file upload will do what they expect, and the system will process the valid rows even if one or more errors are found.

Performance limitations

“Data Import” Workflow Steps

The foundation of the Workflow Import Queue is a new type of Workflow Step called a “Data Import Step”.

A traditional Workflow Step, now called a “Data Entry” Step, describes the form that is displayed to a user within LearningBuilder to collect data at a specific part of a multi-part process.

By comparison, Data Import Steps describe the format for a data file that can be uploaded to create or modify data regardless of the step the entity is currently “on”. In other words, Data Import Steps describe an “out-of-band data manipulation” that can be performed via file upload.

For more information on Data Import Steps, see https://heuristicsolutions.atlassian.net/wiki/spaces/DOCS/pages/1374584869

Supported file types

As of 10.4.0, the following types of data can be imported using the Workflow Import Queue:

CSV file conventions

The specific columns that are required / supported will vary according to the type of file being uploaded, but there are some conventions that common to all file types:

• Value columns

• Identifier columns (“:” prefix)

• Comment columns (“#” prefix)

• Identifying the target Workflow Step / Workflow Action

Value Columns (no prefix)

Value columns contain the values that are used to insert or update data into LearningBuilder. Column headers without a special prefix are value columns.

Typically, each Attribute on the target Data Import Step should have a corresponding value column in the uploaded file.

Identifier Columns (“:” prefix)

Columns that begin with a colon, such as :UniqueId, are identifiers. These columns are used to identify the existing record that should be updated (if it exists) or used to create a new record (if no matching record exists).

This convention allows the system to update the unique identifiers associated with a record. For instance, in a system that uses Unique Id as a unique identifier for a Member Role, a file can change an existing Unique Id like this:

Open

See the individual upload feature for a list of the specific identifiers supported for each file type.

Comment Columns (“#” prefix)

Columns that begin with a hash symbol, such as #Notes, are comments. These columns are ignored by the processing queue.

Typically, the import queue will warn users when they upload a file containing unrecognized columns. This convention allows users to put notes or other metadata into their file without triggering that warning.

Open

Identifying the target Workflow Step / Workflow Action

Each row in an uploaded file must map to a specific Workflow Action on a Data Import Step. This Action will be executed as part of the file upload and can be used to trigger additional, custom behavior in response to the upload.

All file types support special identifier columns that specify the Step and/or Action:

These columns are optional in certain circumstances. Depending on the Workflow configuration, one or both of these columns can be omitted or left blank.

Conditional Behaviors

An uploaded file might contain data representing numerous types of situations. For instance, some rows might represent “passing” exam scores, others might represent “failing” scores. The actions that the system takes in response to a passing score might be very different than those it takes for a failing score, and the Workflow Import Queue supports this level of configurability through Conditional Behaviors.

For more information, see https://heuristicsolutions.atlassian.net/wiki/spaces/DOCS/pages/1565065245

Known Limitations

Date Format must Match Acceptable Database Formats

CSV files have no way of indicating the format used for a date. Although there are helpful options in Excel that can interpret the date format, the file itself when encoded as comma delimited does not indicate a date format. Thus, the workflow import queue has no way of knowing if, for example, “01-02-2000” is written in MM-DD-YYYY format or DD-MM-YYYY format. Instead, it interprets the date based on the date formats read within the database.

“Warnings” are not enforced on uploaded files

Workflow Validation Rules can be configured as “errors” or “warnings”. In the Workflow Popup, if the user enters data that triggers a warning, they are shown the message but are given the option of ignoring it and saving the data anyways.

The Workflow Import Queue will enforce errors, but ignores validation rules configured as a “warning”. At this time there is no UI support for displaying the warnings to the user and asking for an override, so the import queue ignores the warnings.