Skip to main content
Skip table of contents

Configuring the multi-threaded publishing object queues

As of the Data Bridge 3.3 release, you can configure settings for the multi-threaded object queues used for Kafka and file publishing (including streaming exports and snapshots) in Data Bridge System Admin Console.

Object queues are a concurrency control mechanism which supports sending and receiving internal messages between application components. Object queues provide a clean interface for managing flows of integration messages or of asynchronous business logic invocations.

The following snapshot shows an example of a Data Bridge object queue in Data Bridge System Admin Console:

Data Bridge object queue example

The following table describes the object queues that can be configured for Data Bridge:

The Concurrency field value is the maximum number of messages in the queue that can be in the Processing state at the same time.

The Error Count field value is the number of attempts that can be made before the queue stops executing or delivering a message. Messages that reach this error count are delivered to an error queue.

The Global field value indicates whether or not the queue concurrency is applied across the entire application cluster (True) or per node (False).

Object queue nameUsage descriptionConcurrency
(model default)
Error Count
(model default)
Global
brg:BulkExportQueue

An internal queue for UI requests for delimited file, JSON file, or Kafka snapshots that contains one message for each request.

41False
brg:BulkExportErrorQueue

An error queue for messages that fail in the BulkExportQueue, which are generated when delimited file, JSON file, or Kafka snapshot requests fail when being broken down to multiple threads.

Data Bridge doesn't currently have any processes that consume these messages, which means that this object queue's concurrency value is not used as no threads are triggered.

Not applicableNot applicableFalse
brg:ExportCountQueue

An internal queue that processes UI requests to refresh the count of a given ViewDefinition.

43False
brg:ExportStatusQueue

An internal queue to ensure the synchronization of updates to the ExportStatus objects that is used only for bulk exports. There is one ExportStatus object for each bulk export request but multiple threads perform the actual export.

Each bulk export has the following messages:

  • One message for each thread to update the ExportStatus counts
  • One Sentinel message to set the ExportStatus status

This queue also updates the statistics and history records in the Data Bridge UI.

11True
brg:ChunkedBulkExportProcessQueue

This object queue is used to for the orchestration of concurrent snapshot publishing processes whenever the "chunked" / multi-threaded option is enabled. It is meant to have multiple (default 8) processing threads attached to it on each Data Bridge application server node. Each message on this queue typically represents a page of data to be processed.

Error messages are routed to the ObjectErrorQueue.

81False
brg:BulkExportProcessQueue

This object queue is used to for the orchestration of sequential snapshot publishing processes whenever the "chunked" / multi-threaded option is disabled. By design, it should only have a single processing thread attached to it across the entire Data Bridge application cluster. Each message on this queue typically represents a page of data to be processed.

Error messages are routed to the ObjectErrorQueue.

11True

Data Bridge supports per-node concurrency for object queues that allows concurrency to be a factor of the number of server nodes. This means that the concurrency value can grow as the number of nodes increases, rather than you having to manually increase the value as more nodes are added. A boolean property named "global" has been added to object queue channels that determines how the queue concurrency configuration is interpreted. For example, if a queue’s configured concurrency is 8 and global is set to false, the run-time concurrency for the queue on a 4-node cluster is 32. Generally, a queue should be configured as per-node, unless concurrency has a hard limit (for example, an external service to which the messages connect can only accept a fixed number of connections from Data Bridge).

The global property of existing object queues will be set to true to maintain current behavior. In newly created object queues, it will default to false and can be explicitly set to true by updating the channel metadata.

Once the queues have been created, subsequent changes to the global property can be made through database update scripts against the OQObjectQueue table, which is in the ObjectQueueDatabase data source.

To edit settings for an object queue:

  1. In Data Bridge System Admin Console, navigate to the Object Queues page.
  2. Select an object queue from the Object Queues list.
  3. In the details zone, select the Detail tab.
  4. Click EditThe Object Queue properties dialog opens.
  5. Make any of the following changes:
    1. In the Concurrency field, enter a new value to define the maximum number of messages that can be in the Processing state at the same time. A value of 0 indicates unlimited concurrency.

      If you change queue concurrency settings, ensure that you do not adversely affect other queues and the Object Queue engine. For example, consider factors such as the priority of the queue, expected load on the queue, and the maximum receivers setting for the Object Queue engine.

    2. Select Send Enabled to allow the object queue to send messages.
    3. Select Receive Enabled to allow the object queue to receive messages.
  6. Click OKThe Object Queue properties dialog closes.


JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.