Configuring the multi-threaded publishing object queues
As of the Data Bridge 3.3 release, you can configure settings for the multi-threaded object queues used for Kafka and file publishing (including streaming exports and snapshots) in Data Bridge System Admin Console.
The following snapshot shows an example of a Data Bridge object queue in Data Bridge System Admin Console:
The following table describes the object queues that can be configured for Data Bridge:
The Concurrency field value is the maximum number of messages in the queue that can be in the Processing state at the same time.
The Error Count field value is the number of attempts that can be made before the queue stops executing or delivering a message. Messages that reach this error count are delivered to an error queue.
The Global field value indicates whether or not the queue concurrency is applied across the entire application cluster (True) or per node (False).
| Object queue name | Usage description | Concurrency (model default) | Error Count (model default) | Global |
|---|---|---|---|---|
| brg:BulkExportQueue | An internal queue for UI requests for delimited file, JSON file, or Kafka snapshots that contains one message for each request. | 4 | 1 | False |
| brg:BulkExportErrorQueue | An error queue for messages that fail in the BulkExportQueue, which are generated when delimited file, JSON file, or Kafka snapshot requests fail when being broken down to multiple threads. Data Bridge doesn't currently have any processes that consume these messages, which means that this object queue's concurrency value is not used as no threads are triggered. | Not applicable | Not applicable | False |
| brg:ExportCountQueue | An internal queue that processes UI requests to refresh the count of a given | 4 | 3 | False |
| brg:ExportStatusQueue | An internal queue to ensure the synchronization of updates to the ExportStatus objects that is used only for bulk exports. There is one ExportStatus object for each bulk export request but multiple threads perform the actual export. Each bulk export has the following messages:
This queue also updates the statistics and history records in the Data Bridge UI. | 1 | 1 | True |
| brg:ChunkedBulkExportProcessQueue | This object queue is used to for the orchestration of concurrent snapshot publishing processes whenever the "chunked" / multi-threaded option is enabled. It is meant to have multiple (default 8) processing threads attached to it on each Data Bridge application server node. Each message on this queue typically represents a page of data to be processed. Error messages are routed to the ObjectErrorQueue. | 8 | 1 | False |
| brg:BulkExportProcessQueue | This object queue is used to for the orchestration of sequential snapshot publishing processes whenever the "chunked" / multi-threaded option is disabled. By design, it should only have a single processing thread attached to it across the entire Data Bridge application cluster. Each message on this queue typically represents a page of data to be processed. Error messages are routed to the ObjectErrorQueue. | 1 | 1 | True |
Data Bridge supports per-node concurrency for object queues that allows concurrency to be a factor of the number of server nodes. This means that the concurrency value can grow as the number of nodes increases, rather than you having to manually increase the value as more nodes are added. A boolean property named "global" has been added to object queue channels that determines how the queue concurrency configuration is interpreted. For example, if a queue’s configured concurrency is 8 and global is set to false, the run-time concurrency for the queue on a 4-node cluster is 32. Generally, a queue should be configured as per-node, unless concurrency has a hard limit (for example, an external service to which the messages connect can only accept a fixed number of connections from Data Bridge).
The global property of existing object queues will be set to true to maintain current behavior. In newly created object queues, it will default to false and can be explicitly set to true by updating the channel metadata.
Once the queues have been created, subsequent changes to the global property can be made through database update scripts against the OQObjectQueue table, which is in the ObjectQueueDatabase data source.
To edit settings for an object queue:
- Make any of the following changes:
In the Concurrency field, enter a new value to define the maximum number of messages that can be in the Processing state at the same time. A value of 0 indicates unlimited concurrency.
If you change queue concurrency settings, ensure that you do not adversely affect other queues and the Object Queue engine. For example, consider factors such as the priority of the queue, expected load on the queue, and the maximum receivers setting for the Object Queue engine.
- Select Send Enabled to allow the object queue to send messages.
- Select Receive Enabled to allow the object queue to receive messages.