Push notification architecture overview

Push notifications in NexJ CRM rely on two components: the push server and push redirector.

This information provides a brief summary of each of the components as well as an overview of the process that occurs when a client (browser) subscribes to push notifications. The image below shows the push notification component architecture.

Info

For information on deploying push notifications, see (24.08) Configuring push notification#pushserverconfig.

Push server

The push server is a servlet deployed within the NexJ Model Engine.

The push server receives initial client requests, authenticates them, and redirects them to the push redirector.

The push server maintains a persistent HTTP connection with the push redirector to allow it to serve as the API entry
point for the metadata layer and to allow push notifications to be sent to the client through the push redirector.

Push redirector

The push redirector is a servlet deployed on NexJ Model Server.

The application server is based on Apache Tomcat and allows the push redirector to take advantage of a special Tomcat interface called CometProcessor. CometProcessor provides Comet capability with Java NIO.

The push redirector establishes a regular HTTP connection with the push server and listens for notifications coming back on the connection. It also establishes a Comet connection with clients that have been authenticated and redirected by the push server, allowing them to receive push notifications on the connection.

The push redirector is deployed alongside a push server running on a J2EE application server. The push redirector cannot be deployed behind a web server. Direct connections to the push redirector are required to avoid scalability issues. Web servers do not support Java NIO and Comet.

Notification protocol

A notification either contains a key that points to data in the system or it contains the data payload, depending on the fault tolerance requirements of the specific notification.

Key notifications are fault tolerant because they look up data in a master record on the server. Data payload notifications are not fault tolerant because notifications can potentially be lost.

When a client initially registers for push notifications, it specifies the keys for the notifications it wants to receive. If no key is specified, the client receives all notifications targeted to that client, regardless of a notification's key.

When a client receives a key notification, the client queries another master data source for the data corresponding to the notification key it received. The process of querying for this data is client specific. Additionally, it is expected that the client queries the master data source immediately after registering for push notifications to ensure that it has the latest data.

With this design, the client is not dependent on receiving notifications to obtain the latest data. If a notification occurs before a client is registered for push notifications, or during the time that a client is connecting or registering again, the client will not receive stale data. This design increases the reliability of notifications by preventing missed notifications.

Notification modes

Notifications are sent using two modes: raw data and a JavaScript callback function.

Clients can decide at run time which method to use depending on which client-side technique is used to receive and process the push notifications.

Raw data

Notifications are written directly to the output stream of the HTTP response in the following format: <notificationlength>:<notification>. Subsequent notifications are appended to the output stream.

This mode is intended to be used by the XMLHttpRequest (XHR) JavaScript object on the client side. The XHR object does not have access to the raw output stream of the response and can see the entire server output only by examining the responseText property. The recommended approach for querying responseText is to query periodically for new data to keep track of what was already read and to ensure that only new data is read on each iteration.

Info

The XHR cannot follow cross-domain redirects. To avoid this limitation, when a client sends the initial request to the push server to register for push notifications, the push server outputs the redirect URL in its response instead of performing an HTTP redirect to the push redirector. It is the responsibility of the client to send the second request to the push redirector URL.

The example below shows client-side code that uses the raw data notification mode.

<html>
  <head>
    <script type="text/javascript">
      // Initial offset to skip dummy data needed b/c of IE bug.
      var lastPos = 256;
      var xhr;
 
      if (typeof XDomainRequest != "undefined") // IE
      {
        xhr = new XDomainRequest();
      }
      else // Non IE
      {
        xhr = new XMLHttpRequest();
      }
      function load()
      {
        xhr.open("get", "push", true);
        xhr.send(null);
      
        /*
         * The output of the Push Server is the URL to the Push
         * Redirector (manual redirect) to avoid cross domain issues.
         */
        xhr.open("get", xhr.responseText, true);
        xhr.send(null);
        setInterval(poll, 100);
      }

      function poll()
      {
        var res = xhr.responseText.substring(lastPos);
        var pos = 0;
        var len = "";


        while (res.charCodeAt(pos) > 47 && res.charCodeAt(pos) < 58)
        {
          len += res.charAt(pos++);
        }
        var readLen = parseInt(len) + len.length + 1;


        if (res.length >= readLen)
        {
          lastPos += readLen;
          document.getElementById('content').innerHTML =
            res.substring(len.length + 1, readLen);
        }
      }
    </script>
  </head>
  <body onload="load()"><div id="content"/></body>
</html>

JavaScript callback function

The client specifies a JavaScript callback function in a URL parameter when establishing the initial Comet connection. The function must have a single parameter.

Notifications are written to the output stream of the HTTP response as the invocation of the JavaScript function with the notification as the parameter passed to the function:
<script type='text/javascript'<function>('<notification>');</script>

This mode is intended to be used in an IFrame so that the server output is rendered directly by the browser and the function is executed on demand as the output is rendered. The function can use the notification passed to it to set other fields on the page, including fields in a different IFrame or in the parent document.

Info

Because of cross-domain security restrictions, both the IFrame accessing the push server or push redirector and the parent where the IFrame is declared must share a common ancestor domain. The document.domain variable must also be set to the common ancestor domain in the IFrame’s parent.

The example below shows client-side code that uses the JavaScript callback function mode.

<html>
  <head>
    <script type="text/javascript">
      // Allow access to cross-domain URLs with a common ancestor domain
      document.domain="<common-domain>";

      function update(value)
      {
        document.getElementById('content').innerHTML = value;
      }
    </script>
  </head>
  <body>
    <div id="content"/>
    <iframe src="push" frameborder="0/>
  </body>
</html>

Comet modes

Data streaming and long polling Comet modes are supported. The client can decide at run time which mode to use depending on which mode works best with the particular technology stack or network topology.

Info

For key based notifications, the client should request the latest data from the master data source to ensure that the client does not hold stale data due to missed notifications.

Data streaming

When the client requests information from the server, the server holds the request indefinitely. As information becomes available to the server, it is pushed to the client by writing it to the output stream of the response continuously.

Data streaming is recommended and used by default if a mode is not specified by the client. Data streaming cannot be used if there is a proxy server between the client and push redirector that does not handle uncommitted server responses for each request and response. If data streaming is not appropriate, the client must request the long polling mode when the client establishes the initial Comet connection.

Long polling
When the client requests information from the server, the server holds the request until a notification is ready to be pushed to the client. After information becomes available to the server, a completed response is pushed to the client and the connection closes. The client immediately registers for push notifications again in order to continue receiving updates.

Client configuration properties

The client can set the following URL parameters when sending the initial request to register for push notifications. All parameters are optional. If not specified, default values are used.

key
Specifies one or more notification keys that the client is interested in receiving notifications for. If not specified, the client receives notifications for all keys. The parameter can be specified multiple times. For example, key=<key1>&key=<key2>...&key=<keyN>.

mode
Specifies the Comet mode for the client. Valid values are 0 for data streaming and 1 for long polling. If not specified, data streaming applies by default.

callback
Specifies the name of JavaScript callback function to invoke. The notification is passed in as the value for the parameter. For more information, see JavaScript callback function.

If this parameter is not specified, by default the raw data notification mode is used.

Push notification workflow details

This section describes how the client, web server, push redirector, and push server interact during the push notification process.

The following image shows architectural information about the components in the push notification process.

The following sequence diagram shows how the components interact in the push notification process. The numbers in the diagram refer to the steps below that describe the push notification process.

The following steps describe how the client, web server, push redirector, and push server interact during the push notification process:

1. The push redirector establishes a connection with the push server:
   • On startup, the push redirector running on Model Server sends an HTTP request and establishes a standard HTTP connection to the push server running on each node of the application server as part of the NexJ application. The push redirector sends the request using the credentials of a special push user.
     This user is authenticated using the authentication method configured for the NexJ application. The push server authorizes the push redirector by ensuring that the push user has the appropriate push privilege.
   • To hold the connection open indefinitely, the response to the request is not committed by the push server.
   • If a connection failure or timeout occurs, the connection is reestablished. If the push redirector is unable to connect, a connection attempt is repeated every 10 seconds.
     You can configure several push redirectors to connect to the same push server. The push server load balances client requests across the push redirectors (see step 5). If a connectivity disruption occurs, the push redirector reconnects to the push server and sends a count of currently connected clients to the push server. The push server’s load balancing algorithm uses the count to load balance client requests.

2. A client (browser) sends a request to the push server to request and register for push notifications. For example, a browser sends a request through an HTTP server, such as Microsoft Internet Information Services (IIS) or Apache HTTP Server. The client request is authenticated using the authentication method configured for the NexJ application.
   
   If a connection between a push redirector and a push server does not yet exist, the push server waits for up to 60 seconds for the connection to be established. This is to ensure that, if the push redirector and the push server are started at the same time, the redirector has enough time to establish an initial connection to the push server.

   

   Info

   If there is no connection to the push redirector, the push server responds with an HTTP 500 error code. The push server generates a unique client ID and sends it along with the object ID (OID) of the authenticated user to the push redirector on the open connection established earlier. This ensures that only client requests from authenticated users are accepted by the push redirector after redirection by the push serv

3. After receiving the unique client ID and the OID of an authenticated user, the push redirector stores the mapping of the IDs and waits for a maximum of 30 seconds to receive a client request with the same client ID (see step 4). If the request is not received, the client ID and OID mappings are removed from memory. If the client request is received after the redirection timeout period, the request is rejected with an HTTP 403 error (see step 5).

4. The push server responds to the original client request with a redirect, passing the unique client ID as a parameter in the redirect URL:

   

   Info

   The redirect functionality in this case is not a standard HTTP redirect. Instead, the push server simply writes the push redirector URL in the response and the client must follow the redirect URL manually. This is done to avoid cross-domain redirection restrictions. For more information, see Raw data.

   • If multiple push redirectors are available, the push server redirects the client to the push redirector with the lowest number of connected clients. Client count updates are received from the push redirector periodically (see step 8).
   • If the connection to any of the push redirectors is disrupted, the push server takes the push redirector out of rotation and stops redirecting clients to it. The push redirector can establish the connection at a later time if it recovers from the problem that caused the disconnection.
5. The client follows the redirect to the push redirector with the unique client ID in a URL parameter:

   • Upon receiving the request, the push redirector tries to match the client ID from the request to one of the client IDs sent to it previously by the push server. If successful, the push redirector sends a client count update to the push server (see step 8) and establishes a Comet connection with the client. The response is not committed.

     

     Info

     Microsoft Internet Explorer has a limitation where the browser begins flushing the server output of an uncommitted response only after the first 256 bytes have been received. To avoid this limitation, the push redirector writes 256 bytes of dummy data to the response output stream for every new Comet connection that is established with a client. Client-side code should ignore the first 256 bytes of data received from the server for each new connection established with the push redirector. The client can also use the dummy data to detect whether notifications are not reaching it and attempt to take corrective actions, such as switching from data streaming to long polling mode.

   • If the client request arrives at the push redirector before the unique client ID (see step 3), the redirector caches the client request and waits for a maximum of 30 seconds for a matching client ID to arrive from the push server. If the client ID is not received within this time, an HTTP 403 error is returned to the client.
   • If the connection between the push redirector and the push server is disrupted:
     • The push redirector disconnects clients currently connected in 50 ms intervals. This ensures that multiple clients do not attempt to reconnect at the same time.
     • The clients register for push notifications with the push server again.
     • The push server detects the connection disruption and fails clients over to another push redirector.
     • If the push redirector re-establishes its connection with the push server, the push redirector stops disconnecting clients. Connected clients continue to function normally.

6. The push server sends a notification for a specific notification key, when new data in the NexJ application is available to communicate to clients:

   

   Info

   The notification key is required.

   • The notification can be targeted to a specific user OID. If no OID is specified, the notification is broadcast to all clients.
   • If multiple push redirectors are available, the notification is broadcast to all push redirectors because it is not known which push redirector holds the client connection.
7. When the notification is received by the push redirector, it selects clients to push the notification to:
   • If the OID is specified, the push redirector identifies client IDs that are mapped to the OID. If the OID is not specified, all registered clients are selected.
   • Clients that are selected must also be registered to receive notifications for either the specific notification key or for any key.
   • Once all the matching clients are identified, the push redirector sends the notification to clients using the open Comet connection.
8. When a client connects to or disconnects from the push redirector, the push redirector updates the push server with the count of connected clients. To avoid extraneous communication between the redirector and server, the update is not sent on every client connect or disconnect. An update is sent when the actual number of connected clients differs by more than 10 from the number that was last reported to the push server. This optimization can cause load balancing between the push redirectors to be off by up to 10 clients.

Push server and redirector failover design

The push server and push redirector were created with a failover design that, in the event of a lost connection between a redirector and push server, helps prevent dropped client requests, and maintains load balance across redirectors.

Push server failover

When a problem occurs that disrupts the connection between a push redirector and a push server, the push server removes the push redirector from the list of available redirectors and stops redirecting clients to it.

If other redirectors are still available, the server will continue to redirect clients to only those redirectors that are still connected to it. Otherwise, the push server responds to the client with an HTTP 500 error.

The unavailable push redirector can re-establish the connection to the push server when it recovers from the problem that caused it to disconnect.

Push redirector failover

When the push redirector loses its connection to the push server, the redirector begins to disconnect clients that are currently connected to it. This allows the clients to authenticate for push notifications with the push server again. The push server can then redirect the clients to another push redirector.

To prevent the push server from receiving too many client authentication requests at once, the push redirector disconnects clients in 50 millisecond intervals.

If the push redirector establishes a connection to the push sever during this process, it stops disconnecting clients. Clients still connected to the push redirector will continue to function normally, and the push redirector can begin to receive new client requests again.

Push client failover

As of NexJ CRM 9.4, the following information applies:

• When a client cannot establish a push connection within 30 seconds, polling mode is initiated.
• When a client using a push connection receives an error, it attempts to reconnect the push connection with an exponential backoff timeout. The timeout doubles each occurrence (starting at approximately 1 second) to a maximum of 5 minutes. While retrying to connect to push, polling also occurs (as indicated in the client’s About NexJ CRM dialog). Should polling also receive errors, it retries at the same timeout interval as for push. Polling stops when a push connection is resumed.

Communication protocols

The following table describes communication between the components involved in push functionality.

Determining push notification information for your deployment environment

The About NexJ CRM dialog provides information (push URL, state, and retry timeout) about the push connection used for NexJ CRM.

The Push State field can display following values. 

• Starting
• Connecting
• Stopped
• Running (poll in numberOfSeconds)
• Running (push)
• Retrying (push in numberOfSeconds)
• Pending

The Pending state indicates that the push connection is still pending because the initial connection response is buffered, it is not receiving any push payloads, or it is not receiving a heartbeat response from the push redirector.