Synchronization Engine
The synchronization engine provides the ability to synchronize records across multiple system. For example, the engine synchronizes the task and schedule items in NexJ CRM with Microsoft Exchange. Because this use case is so common, it will be used in the following discussion to illustrate synchronization engine concepts.
Purpose of the engine
NexJ's applications are based an object model, usually mapped to an underlying database. Some of the data represented in the object model is generated by users interacting with the application directly. However, much more of the object data typically originates in other systems on a company's network. NexJ CRM needs to be able to read information from these systems in order to register and react to events from these systems, or to notify these systems of events within NexJ CRM and provide data to them for processing or storage.
All of this communication is done by means of messages. The NexJ integration framework provides a library of tools for defining how NexJ applications should create, receive, process, and send messages to the other applications with which it must communicate. Most of this functionality is accessible through specialized UI in NexJ Studio.
Messages
The format and structure of the messages that NexJ applications use depends on the external systems they interact with. Within the NexJ application model and associated code, however, those message formats can largely be ignored; internally, NexJ applications represent messages as Transfer Objects. It is only when a message is sent or received that the format becomes a key issue.
The message type editor looks like this:
Object messages
The Object message format deserves particular attention since it is the foundation for how NexJ applications synchronize with other systems. Each object message represents an object instance. Generally, each object message type is associated with a class, and contains a subset of the attributes of that class.
When an object message is received and turned into a transfer object, that object directly manipulates the object model. That is, if the system receives such a message, then:
- If an object with the same key as the message already exists, then that object is loaded and modified to match the message (an update)
- If an object does not already exist with the same key as the message, then one is created.
The framework also provides a mechanism for generating object messages from object instances. In this case, an object instance is parsed into a transfer object. The object is then converted to a message and sent to external systems in a format that those systems can process.
Channels
Ultimately, messages must be sent or received to be of any value. At design time, the exact sources and destinations of messages are not usually known. However, the required formats and expected transport protocols (HTTP, UDP, etc.) are known. In NexJ CRM, these communication lines are modeled abstractly as Channels. Each channel abstractly defines a message source or destination; that is it captures the transport information, but not the connection parameters. Services can be configured to send messages and receive responses on channels, while Service bindings specify which integration services should process any asynchronously received messages.
Here is the channel editor:
Only when the application is deployed must the connection parameters for all channels be specified:
Transformations
Transformations, as the name suggests, transform instances of one message type to another. A normal synchronization scenario requires two message types: one object message type, very closely matching a class; and one message type with a structure and format understood by the external system. The application must be able to translate from one format to another and back.
Services
Deciding which messages to process, which to ignore, where to send them, how to transform them, and so on, is managed through specialized workflows called Services. Each service has a state, stored in a variable called this. The state is a working copy of the message as it is transformed, parsed, formatted, and so on.
Here's a portion of one of the exchange sync services:
Synchronization Classes
Synchronization Engine
The dirty work takes place at the point where we either format an object message (and merge it into our object model) or decide that an object message must be dispatched on a service for consumption by an external system.
There are two situations of interest. In the first case, we have received a message from an external system and converted it into an object message. We now wish the message to be formatted into an object and merged with our data model. This is usually done with a synchronize step in a service. For each message entering a synchronization step, we try to find a previous instance of the corresponding object. If one is found, then we check to see if there have been any changes to the object since the last time the external system received a copy of it (or gave us a copy). If there have been changes, then we determine on which attributes there are conflicts. Once the object has been found (or created) and any conflicts have been located, all this information is passed to the script of the synchronize step. The script allows an application to "veto" changes proposed by the synchronization engine, and to make decisions about how to resolve any conflicts.
In the second case, a change has been made to data in our object model. During commit of this change, the framework detects that an instance of a class marked for synchronization is being committed, finds interested external systems, builds an object message, and dispatches it to the services configured for all interested external systems.
Both of these scenarios imply that we have some "meta" knowledge about business objects in our model, and their relation to external systems. Today, I will (very briefly) outline the nature of this knowledge.
SysSyncTarget
Every external system with which we synchronize is modeled as a synchronization target. A synchronization target specifies the service to which to send outbound messages, and a bunch of configuration parameters. In the Exchange synchronization code, we have a subclass of SysSyncTarget on which we put additional information, such as the channels on which to send and receive Exchange messages.
SysSyncClass
We relate business logic classes to targets through subclasses of SysSyncClass. For each class we want synchronized with external systems, and for each external system interested in that class, we create one instance of SysSyncClass. SysSyncClass is an interface with a bunch of hooks for customizing the behaviour of the synchronization framework when dealing with instances of the corresponding business class.
SysSyncLink and SysSyncLinkClass
Data storage systems are usually compartmentalized in some way. You don't just dump records in one bin, but rather into many different files, folders, tables, etc. A SysSyncLink represents one of these abstract compartments in a SysSyncTarget. This class is usually subclassed to provide storage for attributes specific to the external system involved.
Every link of a target may not be interested in every class. A SysSyncLinkClass instance serves to associate one or more SysSyncClass instances with a link. Many links will likely share the same instance(s) of SysSyncClass, but most links will have very few associated classes (often only one.)
SysSyncObject and SysSyncGroup
In order to determine whether an external system has seen a copy of any particular instance in our object model requires us to track information about the most recent version of each object sent to or received from each external link. This information is tracked in SysSyncObject instances. Usually there is one sync object per synchronized instance, per link to which it is synchronized.
In some cases one NexJ instance will be synchronized with multiple instances on an external system. The prime example of this is a meeting: we have one copy of the meeting, but Microsoft Exchange stores one copy of the meeting in the mailbox of each participant. In this case, we need to keep track of the group of sync objects corresponding to our one NexJ instance. This is done with an instance of SysSyncGroup.
Unlike the other classes mentioned above, SysSyncObject and SysSyncGroup are not generally subclassed by an application.