Latest improvements for QIAGEN CLC Genomics Cloud Engine

This update contains changes to the infrastructure as well as the software. Due to this, it is necessary to redeploy the CLC Genomics Cloud Engine when upgrading to this version.

Bugfixes

• Fixed a bug affecting CLC Genomics Cloud Engine 20.0.10 that caused it to stop functioning when the CLC License Server was restarted.
• Fixed an issue where tags were not handled correctly if more than one tag was defined during stack creation.
• Fixed an issue that caused a “max depth exceeded” error, when plugins were updated more than 125 times.

Improvements

• The Stockholm (eu-north-1) and the Seoul (ap-northeast-2) regions are now supported.
  
• Support for configuring an OAuth2 resource ID, needed for compatibility with some third party authentication servers, has been added.
  
• The newer db.t3.small instance type is now used instead of the t2 variant.

Frequently Asked Questions

QIAGEN CLC Genomics Cloud Engine Command Line Tools 20.0.16

Bugfixes

• Fixed an issue related to job submission, where S3 locations accessible to CLC Genomics Cloud Engine were being unnecessarily signed in auto-signing mode.

Compatibility

Workflow compatibility

CLC Genomics Cloud Engine 20.0.16 uses CLC Genomics Server 20.0.4.

Workflows to be run on the CLC Genomics Cloud Engine 20.0.16 are designed using a CLC Workbench. Workflow installer files should be made using a CLC Workbench compatible with the CLC Genomics Server run by GCE. The recommended versions for this purpose for this version are:

• CLC Genomics Workbench 20.0.4
• CLC Main Workbench 20.0.4

Workflows can be submitted to run on CLC Genomics Cloud Engine 20.0.16 from compatible versions of CLC Workbenches with the Cloud Plugin installed and from compatible versions of CLC Genomics Server with the Cloud Server Plugin installed.

Service/Client compatibility

While we recommend running corresponding versions of service and client, CLC Genomics Cloud Engine Command Line Tools 20.0.10 & 20.0.16 can be used with CLC Genomics Cloud Engine 20.0.16, and CLC Genomics Cloud Engine Command Line Tools 20.0.16 can only be used with CLC Genomics Cloud Engine 20.0.16.

This update contains changes to the infrastructure as well as the software. Because of the infrastructure changes, it is necessary to redeploy the CLC Genomics Cloud Engine.

New Features

• Batch submission is now supported, allowing CLC GCE users to submit large number of jobs in a single request.
• Files can now be accessed using internal Amazon S3 protocols, which is provided free of charge by AWS. Accessing files via pre-signed URLs is still possible, and should be used when user access rights need to be controlled.
• The distribution of jobs can now be stopped by the administrator. This allows maintenance tasks like changing instance type to be undertaken without stopping job submissions.
• Amazon S3 cache deployment scripts have been introduced. These create one or more cache buckets and are needed to support submission of jobs from CLC Workbenches with the Cloud Plugin installed, or via a CLC Genomics Server with the Cloud Server Plugin installed. These plugins are due for release soon.>
• The OAuth2 Client Credentials flow authentication protocol is now supported.
• Job and Batch search functionality are now available via the REST API.

Improvements

• The validation of job and batch submissions has been substantially improved. This includes validating the version of the CLC Workbench and any associated plugins used to create the workflow.
• A job is no longer launched on GCE in response to requests querying the parameters of a workflow.
• The version of the CLC Genomics Server being used by CLC GCE can now be inspected via the description of the Elastic Beanstalk application version.
• The deleteStack.sh script no longer tries to delete the certificate associated with the JobManager Load balancer.
• All node.js scripts have been upgraded from version 8 to version 10.
• CLC GCE now places results files in subfolders with names that include the job name as well as a unique job ID.
• The maximum number of jobs in a JobManager database has been increased from 100,000 to 10,000,000.

Bugfixes

• Fixed an issue that could sometimes result in QIAGEN reference data not being put in the expected location when downloaded automatically when launching a workflow on the CLC Genomics Cloud Engine. When this happened, workflows depending on that reference data would fail to run to completion. This problem affected CLC Genomics Cloud Engines running CLC Genomics Server 11.0.
• Fixed a timeout error that occurred on the JobManager web pages. Users are now presented with a login dialog if the login has timed out.

Changes

• A new End User License Agreement (EULA) has been introduced.

Frequently Asked Questions

QIAGEN CLC Genomics Cloud Engine Command Line Tools 20.0.10

New features

• Multiple jobs can now be submitted with a single command as a batch.

Improvements

• Querying the parameters of a workflow is now much faster.
• Data from AWS regions other than the one the CLC GCE instance being used is installed upon can now be used.
• GCE CLT no longer pre-signs S3 URLs unless it is is needed for CLC GCE to access the data and automatic pre-signing of S3 URLs has not been disabled in the GCE CLT configuration.
• Instance type and disk size can no longer be changed using GCE CLT. Disk size now scales automatically as needed. Instance type can be changed via the AWS Elastic Beanstalk web console.

Compatibility

The CLC Genomics Cloud Engine Command Line Tools 20.0.10 is the corresponding client for the CLC Genomics Cloud Engine 20.0.10.

We always recommend running the corresponding versions of service and client and for this release in particular it is not possible to access the CLC Genomics Cloud Engine 20.0.10 with an earlier version of the CLC Genomics Cloud Engine Command Line Tools. Nor is it possible to access an earlier version of the CLC Genomics Cloud Engine with the CLC Genomics Cloud Engine Command Line Tools 20.0.10.

The CLC Genomics Cloud Engine uses CLC Genomics Server 20.0.1.

Workflows to be run on the CLC Genomics Cloud Engine 20.0.10 are designed using a CLC Workbench. Workflow installer files should be made using a CLC Workbench compatible with the CLC Genomics Server run by GCE. The recommended versions for this purpose for this version are:

• CLC Genomics Workbench 20.0.1
• CLC Main Workbench 20.0.1

Workflows can be submitted to run on CLC Genomics Cloud Engine 20.0.10 from compatible versions of CLC Workbenches with the Cloud Plugin installed and from compatible versions of CLC Genomics Server with the Cloud Server Plugin installed.

This update contains changes to the infrastructure as well as the software. Because of the infrastructure changes it is necessary to redeploy the QIAGEN  CLC Genomics Cloud Engine.

Improvements

• Release now includes QIAGEN  QIAGEN CLC Genomics Server version 9.0, 10.0.1 and 11.0 and the system administrator can change which of the versions to use.
• Service to service integration is now possible through a convenient REST API. A Swagger Specification is provided with the REST API, which can be used to generate client code for easier integration.
• Improved auto scaling, which scales faster to a large amount of servers and shuts them down faster as well.
• AWS resources used by QIAGEN  CLC Genomics Cloud Engine can now be tagged. The tags are defined during deployment of QIAGEN  CLC Genomics Cloud Engine.
• Improved error messages when a job fails because of file corruption.
• Encryption key moved to database CloudFormation stack. As a result there is no need for re-enrolling users after a redeployment of the other GCE stacks.

Bugfixes

• Job definition files from cancelled job are now deleted when the job is cancelled.
• Unused AWS SQS queues are now deleted.
• “get parameter” jobs are now displayed as successfully completed in the QIAGEN  CLC Genomics Cloud Engine web interface.

Frequently Asked Questions

QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.3.9

Bugfixes

• Changing instance type using the Command Line Tools is now memory efficient. Previously it would sometimes cause an out of memory error on low memory systems.

Compatibility

The QIAGEN  CLC Genomics Cloud Engine Command Line Tools 1.3.9 is the corresponding client for the QIAGEN  CLC Genomics Cloud Engine 1.3.9.

We always recommend running the corresponding versions of service and client and for this release in particular it is not possible to access the QIAGEN  CLC Genomics Cloud Engine 1.3.9 with an earlier version of the CLC Genomics Cloud Engine Command Line Tools. Nor is it possible to access an earlier version of the QIAGEN  CLC Genomics Cloud Engine with the QIAGEN  CLC Genomics Cloud Engine Command Line Tools 1.3.9.

The CLC Genomics Cloud Engine can be run using the QIAGEN  QIAGEN CLC Genomics Server 11.0, 10.0.1 or 9.0.

Workflows to be run on the QIAGEN  CLC Genomics Cloud Engine 1.3.9 are designed using a QIAGEN  CLC Workbench. Workflow installer files should be made using a QIAGEN  CLC Workbench compatible with the QIAGEN CLC Genomics Server used for GCE. Recommended versions for this purpose are listed below. The QIAGEN  CLC Workbench versions that correspond to a given QIAGEN  QIAGEN CLC Genomics Server version are shown in bold font. These are preferred for making workflow installers to be run on GCE.

QIAGEN CLC Genomics Server 11.0

• QIAGEN CLC Genomics Workbench 12.0
• QIAGEN CLC Main Workbench 8.1

QIAGEN CLC Genomics Server 10.0.1

• QIAGEN CLC Genomics Workbench 11.0, 11.0.1
• Biomedical Genomics Workbench 5.0, 5.0.1
• QIAGEN CLC Main Workbench 8.0. 8.0.1

QIAGEN CLC Genomics Server 9.0

• QIAGEN CLC Genomics Workbench 10.0
• Biomedical Genomics Workbench 4.0
• QIAGEN CLC Main Workbench 7.8

This update contains only changes to deployment of the infrastructure configuration.

Bugfixes

• Fixed a deployment issue caused by AWS API changes.

Frequently Asked Questions

CLC Command Line Tools 1.2.4

Nothing has changed.

Compatibility

The Command Line Tools 1.2.4 is the corresponding client for the 1.2.4.

We recommend running the corresponding versions of service and client.However, Command Line Tools 1.2.4 can be used with 1.2.2, 1.1.1, 1.0.15, 1.0.6 and 1.0.4 and Command Line Tools 1.2.2, 1.1.1, 1.0.15, 1.0.6 and 1.0.4 can be used with 1.2.4. includes 10.0.1.

Workflows to be run on the 1.2.4 can be designed using Workbenches compatible with 10.0.1. These are:

• 11.0.1 and 11.0
• Biomedical 5.0.1 and Biomedical 5.0

This update contains only changes to the software. There are no changes to the infrastructure configuration.

Bugfixes

• Fixed a bug that caused an out of memory error when changing the Job Executor instance type using the admin host.

Improvements

• Each AWS resource created during deployment can now be tagged.
• AWS Cognito can be used as the identity provider.
• The QIAGEN CLC Genomics Server included in GCE has been upgraded to version 10.0.1.
• When self-signed certificates are generated they will now be valid for 30 years rather than 1 year.
• The supported AWS regions have been expanded to the following: N. Virginia, Ohio, N. California, Oregon, Frankfurt, Ireland, Tokyo, Seoul, Sydney and Mumbai.

Frequently Asked Questions

QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.2.2

Improvements

• The stability of the “job parameters” command has been improved.

Compatibility

The QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.2.2 is the corresponding client for the QIAGEN CLC Genomics Cloud Engine 1.2.2.

We recommend running the corresponding versions of service and client. However, QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.2.2 can be used with QIAGEN CLC Genomics Cloud Engine 1.1.1, 1.0.15, 1.0.6 and 1.0.4, and QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.1.1, 1.0.15, 1.0.6 and 1.0.4 can be used with QIAGEN CLC Genomics Cloud Engine 1.2.2.

The CLC Genomics Cloud Engine includes the QIAGEN CLC Genomics Server 10.0.1.

Workflows to be run on the QIAGEN CLC Genomics Cloud Engine 1.2.2 can be designed using Workbenches compatible with QIAGEN CLC Genomics Server 10.0.1. These are:

• QIAGEN CLC Genomics Workbench 11.0.1 and QIAGEN CLC Genomics Workbench 11.0
• Biomedical Genomics Workbench 5.0.1 and Biomedical Genomics Workbench 5.0

This update contains only changes to the software. There are no changes to the infrastructure configuration.

Bugfixes

• Jobs will no longer fail during the final phases of performing the analysis because of AWS S3 occasionally failing to respond to requests. GCE now handles this with retries in accordance with the AWS recommended exponential backoff strategy.

Improvements

• The new Administration Host replaces the Installation Host. The Administration Host is installed separately from the “preinstall” stack, via a separate CloudFormation script. To learn more read the “Install GCE on Amazon” section and the new “Upgrade an Existing GCE Solution” section of the Administration Manual.
• Included with the Administration Host is a new script that will update GCE from a previous version to the newest version. To learn more read the new “Upgrade an Existing GCE Solution” section of the Administration Manual.

Frequently Asked Questions

CLC Genomics Cloud Engine Command Line Tools 1.0.15

Bugfixes

• Job submits no longer fail because of AWS S3 occasionally failing to respond to requests. GCE CLT now handles this with retries in accordance with the AWS recommended exponential backoff strategy.

The CLC Genomics Cloud Engine Command Line Tools 1.0.15 is the corresponding client for the QIAGEN CLC Genomics Cloud Engine 1.0.15.

We recommend running the corresponding versions of service and client. However, QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.0.15 can be  used with QIAGEN CLC Genomics Cloud Engine 1.0.4 and 1.0.6, and QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.0.4 and 1.0.6 can be used with QIAGEN CLC Genomics Cloud Engine 1.0.15.

The QIAGEN CLC Genomics Cloud Engine includes the QIAGEN CLC Genomics Server 9.0.

Workflows to be run on the QIAGEN CLC Genomics Cloud Engine 1.0.15 can be designed using Workbenches compatible with QIAGEN CLC Genomics Server 10.0.0 and 10.0.1. These are:

• QIAGEN CLC Genomics Workbench 10.0.0 and 10.0.1
• Biomedical Genomics Workbench 4.0

Information on getting Workbench installers

Bug fixes

• Fixed an issue where log URLs were not returned if a job failed early on.

QIAGEN CLC Genomics Cloud Engine Command Line Tools

Improvements

• The option -profile has been added to several commands, to support the use of multiple GCE setups.
  • gce config service … -profile <profileName>                      to associate a profile name with a particular GCE setup
  • gce config aws … -profile <profileName>                          to configure AWS credentials for a specified GCE setup
  • gce admin autoscaling … -profile <profileName>               to configure autoscaling for a specified GCE setup
  • gce admin executor-settings … -profile <profileName>      to configure the Job Executor of a specified GCE setup
  • gce job … -profile <profileName>                                       to perform job operations on a specific GCE setup
• Paths to destination folders no longer need to end with a “/”.
• The IAM:GetUser AWS permission is no longer required.
• Fetching a list of input parameters for a workflow (gce job parameters) now runs with asap as the default priority. Previously it ran with priority set to medium.

Compatibility information

The QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.0.6 is the corresponding client for the QIAGEN CLC Genomics Cloud Engine 1.0.6.

We recommend running the corresponding versions of service and client. However, QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.0.6 can be used with QIAGEN CLC Genomics Cloud Engine 1.0.4, and QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.0.4 can be used with QIAGEN CLC Genomics Cloud Engine 1.0.6.

The QIAGEN CLC Genomics Cloud Engine includes the QIAGEN CLC Genomics Server 9.0.

Workflows to be run on the QIAGEN CLC Genomics Cloud Engine 1.0.6 can be designed using Workbenches compatible with QIAGEN CLC Genomics Server 9.0 . These are:

• QIAGEN CLC Genomics Workbench 10.x
• Biomedical Genomics Workbench 4.x

Frequently Asked Questions

First release
Frequently Asked Questions

Compatibility

The QIAGEN  CLC Genomics Cloud Engine Command Line Tools 1.0.4 is the client software for the QIAGEN  CLC Genomics Cloud Engine 1.0.4.

The QIAGEN  CLC Genomics Cloud Engine 1.0.4 includes the QIAGEN  QIAGEN CLC Genomics Server 9.0.

Workflows to be run on the QIAGEN  CLC Genomics Cloud Engine 1.0.4 can be designed using Workbenches compatible with QIAGEN CLC Genomics Server 9.0. These are:

• QIAGEN CLC Genomics Workbench 10.0.x
• Biomedical Genomics Workbench 4.0.x

QIAGEN CLC Genomics Cloud Engine Command Line Tools 1.0.4

First release