Monitoring application server cluster health

You can monitor the health of your application server clusters from the System page of NexJ System Admin Console using the diagnostic commands provided at the bottom of the page. They also provide ways to ping all nodes, clear all caches, dump a specific cache value, and clear a specific cache value.

A lack of response to any of the diagnostic commands from a node indicates a problem. For example, the message may have been lost or the node may be down.

The DiagnosticsManage privilege is required to use these diagnostic commands. The INFO logging level is also required.

For more information about configuring logging levels for NexJ Model Server, see Updating logging levels. For information about configuring the logging level for WebSphere, see Monitoring application server cluster health in the system administration documentation for the 8.9 release.

Using the Ping All Nodes command

Use the Ping All Nodes command to ensure that all nodes in a cluster will log additional information about themselves. This option broadcasts a request for all nodes in the cluster to print an informational message to the log. Healthy nodes will write some basic statistics to the log such as their ID, name, and CPU and memory utilization. In the log, one line will indicate when the broadcast request was sent and one line should appear per node as they respond.

Sample logged request and response

In this example, the Ping All Nodes action is used against a two-node cluster.

Request:

CODE

[6/23/14 11:45:46:115 EDT] 00000054 diagnostics I [http:192.0.2.0:34659
admin@EXAMPLE.LOCAL] Broadcasting ping request

Response from Node 1:

CODE

[6/23/14 11:45:46:123 EDT] 00000061 diagnostics I [ObjectSystemQueue
admin@EXAMPLE.LOCAL] ---Ping--- #<Node(id=L94QqqgfQG2ytB00vcU/6w==,
httpNode=18d2fgkj4, cpu=5%, memory=25%)>

Response from Node2:

CODE

[6/23/14 11:45:46:131 EDT] 0000002c diagnostics I [ObjectSystemQueue
admin@EXAMPLE.LOCAL] ---Ping--- #<Node(id=MsI7R0RtR3eCdpNs1dGgBg==,
httpNode=18d2fgr7d, cpu=2%, memory=23%)>

Using the Clear All Caches command

Use the Clear All Caches command to broadcast a request for all nodes in the cluster to clear their cache by invalidating all cache items. You will receive a confirmation dialog before the request is sent.

Sample logged request and response

In this example, the Clear All Caches action is used against a two-node cluster.

Request:

CODE

[6/23/14 11:46:02:286 EDT] 00000054 diagnostics I [http:192.0.2.0:34661
admin@EXAMPLE.LOCAL] Broadcasting clear cache value request for key: ()

Response from Node 1:

CODE

[6/23/14 11:46:02:293 EDT] 00000046 diagnostics I [ObjectSystemQueue
admin@EXAMPLE.LOCAL] Invalidating all items in the global cache

Response from Node2:

CODE

[6/23/14 11:46:02:301 EDT] 0000002b diagnostics I [ObjectSystemQueue
admin@EXAMPLE.LOCAL] Invalidating all items in the global cache

Search key formats

You can specify search keys when using the diagnostics tools using different format options.

The following formatting options are available for specifying the search keys, based on how your code is implemented:

Class name

The following formatting options are available for specifying the search keys, based on how your code is implemented:

FormatClassNameExampleUser
<ClassName>

Example
User

Space-separated class name and binary OID for instance-cached classes

The expression used in the search can only contain Scheme literals. Function calls or expressions that require evaluation are not supported.

Format
<ClassName> <OID> where the OID can either include or exclude the initial 0xcharacters.

Example
User 00000000000010008000BEEF0000000C

Scheme expression for the case when the cache is populated with a Scheme expression

Format
(<ClassName> <attributeName> <Value>)

Example
(User loginName "<ADMIN>")

Attribute names are case-sensitive. The value for loginName is supplied in upper case because this is how users are cached in the system. A lower-case value would not return any data.

Using the Dump Cache Value command

Use the Dump Cache Value command to submit a search key and receive the related value from the cache of each node. The request asks all nodes in a cluster to print out the value for the searched cache key to the logs. The log entries indicate basic information about the cached references. In particular, the Tx timestamp indicates the time when the transaction that added the value to the cache was started.

For more information about search keys, see Search key formats.

In the examples below, only the response of a single node is provided. In practice, you will receive information from each node in the cluster.

Sample logged request and response - null response

Request:

CODE

[6/23/14 12:08:23:750 EDT] 00000059 diagnostics I [http:192.0.2.0:34675
<admin>@<EXAMPLE>.LOCAL] Broadcasting dump cache value request for key: (User
loginName "<CustomerLoginName>@<EXAMPLE>.LOCAL")

Response:

In this example, the null response suggests that the search key instance has not been cached at all. In the case when you are using a user search key, the easiest way to cache the desired instance is to log into the application as the user.

CODE

[6/23/14 11:47:08:140 EDT] 0000006c diagnostics I [ObjectSystemQueue
admin@EXAMPLE.LOCAL] Cached reference for key: (User loginName
"CustomerLoginName@EXAMPLE.LOCAL")
null

Sample logged request and response - node with a valid cached value

Request:

CODE

[6/23/14 12:08:23:750 EDT] 00000059 diagnostics I [http:192.0.2.0:34675
<admin>@<EXAMPLE>.LOCAL] Broadcasting dump cache value request for key: (User
loginName "<CustomerLoginName>@<EXAMPLE>.LOCAL")

Response:

CODE

[6/23/14 12:08:23:765 EDT] 000000a9 diagnostics I [ObjectSystemQueue
<admin>@<EXAMPLE>.LOCAL] Cached reference for key: (User loginName
"<CustomerLoginName>@<EXAMPLE>.LOCAL")
GenericDataCache$Ref@1767926112{key=(User loginName "<CustomerLoginName>@<EXAMPLE>.LOCAL"),
Tx timestamp=2014-06-23 12:06:25.736, value=UnitOfWork
$CachedInstance(TO<InternalUser, OID:1:V32:3ED0B094C84B4634B11044BF8C3192E8>(
entitySet={
OID:1:V32:EAFD401DB8C24B10A45C0DE3BA23E562
},
entityListSet={
OID:1:V32:3ED0B094C84B4634B11044BF8C3192E8
},
categorySet={},
alias="ratesttest",
favouriteUserList=TO<FavouriteUserList,
OID:1:V32:EEC91AC5CFE24B6BBFAE207C4798DB14>(),
loginName="<CustomerLoginName>@<example>.local",
principalSet={
OID:1:V32:00000000000010008000BEEF0000000A,
OID:1:V32:3ED0B094C84B4634B11044BF8C3192E8,
OID:1:V32:5C0D4539BE557A43B28C997E99ADF3AC
},
branchPrincipalSet={},
entitlementsHash=(),
optionBlock=TO<OptionBlock, OID:1:S23:<CustomerLoginName>@<example>.local>(
enableCopyMerge=#t,
timeEnforceMaxDuration=#f,
enableFinancialModel=#t,
enableRegisteredPlans=#t,
enableProducts=#f,
enableDisplayExternals=#f,
enableRecurringTaskOutboundSync=#t,
enableRecurringMeetingOutboundSync=#t,
enableExtendedCustomField=#f,
enablePortalAccess=#f,
enableServiceRequests=#f,
enableRecentDocuments=#t,
enableCampaigns=#t,
enableForcedBatchLogging=#f,
enableForAssignToFixup=#t,
enableMeetingInvitations=#t,
enableCreateFolder=#t,
enableNativeMsgListIntegration=#t,
enableRecurringMeetingInboundSync=#t,
enableMeetingAlarmSync=#t,
enableForContactSync=#t,
enableRecentEmails=#t,
enableTimeZones=#f,
enableLocalization=#t,
enableTaskAlarmSync=#t,
enableTimeTracking=#f,
enableUserFields=#t,
enableUsersAsContacts=#t,
enableRecentScheduleItems=#t,
enableFinancialAccounts=#t,
enableSlaveSync=#t,
forContactSyncWithSecurityCheck=#t,
enableMatchingDuringDataImport=#f,
enableHouseholds=#t,
autoCreateFolder=#f,
enableCanadaAddressLookup=#f,
timeEnforceOnActDay=#f,
enableAutoAssignTeam=#f,
enableInteractiveReports=#t,
enableCoverageQuickSearch=#t,
enableRelatedInfoNotes=#f,
enableOpportunities=#t,
enablePublicSchedTask=#f,
enableRecentTasks=#t,
enableRecentTaskRecurrence=#t,
enableParentCoverageUpdates=#t,
timeWarnDailyLimit=#f,
enableDefaultForContact=#t,
enableRecurringTaskInboundSync=#t,
enableContactTitleSync=#f,
enableRecentScheduleItemRecurrence=#t,
enableEvents=#t,
enableClientSideErrorLogging=#f,
enableRecentActivityPlans=#t
),
person=TO<UserPerson, OID:1:V32:EAFD401DB8C24B10A45C0DE3BA23E562>(
hasSyncList=#t,
classCode="USR",
syncList=TO<SyncEntityList, OID:1:V32:C61E252837154FFEAE265FFD3814EFA8>()
),
userListSet={
OID:1:V32:3ED0B094C84B4634B11044BF8C3192E8
},
privilegeSet=PrivilegeSet@823669016(mask=
0000FFBF93F7FC0F47FC73084CFD97800000124100000000180000802030C000200000208402),
firmPrincipalSet={
OID:1:V32:5C0D4539BE557A43B28C997E99ADF3AC
},
coverageTeamSet={},
aclPrincipalSet={
OID:1:V32:3ED0B094C84B4634B11044BF8C3192E8,
OID:1:V32:5C0D4539BE557A43B28C997E99ADF3AC
},
customFieldTypeSet={}
))}

Sample logged request and response - node with invalidated cache value

Request:

CODE

[6/23/14 12:08:23:750 EDT] 00000059 diagnostics I [http:192.0.2.0:34675
<admin>@<EXAMPLE>.LOCAL] Broadcasting dump cache value request for key: (User
loginName "<CustomerLoginName>@<EXAMPLE>.LOCAL")

Response:

CODE

[6/23/14 12:08:23:768 EDT] 0000002a diagnostics I [ObjectSystemQueue
<admin>@<EXAMPLE>.LOCAL] Cached reference for key: (User loginName
"<CustomerLoginName>@<EXAMPLE>.LOCAL")
GenericDataCache$RemovedRef@885929166{key=(User loginName
"<CustomerLoginName>@<EXAMPLE>.LOCAL"), Tx timestamp=2014-06-23 11:50:08.171,
value=java.lang.Object@5cae5cae}

Using the Clear Cache Value command

Use the Clear Cache Value command to submit a search key and clear its related value from the caches of all nodes in the cluster. The clearing is achieved by invalidating the cache value for the key, ensuring that the cached key value will not be used.

The same set of keys can be searched for and used to clear a cache value as to dump a cache value. For more information, see Search key formats.

Sample logged request and response using user search key

When you clear the cache, the response received the first time differs from subsequent times. The first use of the tool gives a result similar to the following request and response:

Request:

CODE

[6/23/14 12:20:17:742 EDT] 0000007f diagnostics I [http:192.0.2.0:34689
<admin>@<EXAMPLE>.LOCAL] Broadcasting clear cache value request for key: (User
loginName "<CustomerLoginName>@<EXAMPLE>.LOCAL")

Response:

CODE

[6/23/14 12:20:17:748 EDT] 00000046 diagnostics I [ObjectSystemQueue
<admin>@<EXAMPLE>.LOCAL] Invalidating 1 item in the global cache keyed by: (User
loginName "CustomerLoginName>@<EXAMPLE>.LOCAL")

After you clear the cache value, however, a second search on the same key will result in an invalid cache instance with the GenericDataCache$RemoveRef being returned, as shown in the following request and response:

Request:

CODE

[6/23/14 12:21:04:854 EDT] 0000007f diagnostics I [http:192.0.2.0:34691
<admin>@<EXAMPLE>.LOCAL] Broadcasting dump cache value request for key: (User
loginName "<CustomerLoginName>@<EXAMPLE>.LOCAL")

Response:

CODE

[6/23/14 12:21:04:876 EDT] 00000057 diagnostics I [ObjectSystemQueue
<admin>@<EXAMPLE>.LOCAL] Cached reference for key: (User loginName
"<CustomerLoginName>@<EXAMPLE>.LOCAL")
GenericDataCache$RemovedRef@20840766{key=(User loginName
"<CustomerLoginName>@<EXAMPLE>.LOCAL"), Tx timestamp=2014-06-23 12:20:17.749,
value=java.lang.Object@49f249f2}
[6/23/14 12:21:31:988 EDT] 00000082 diagnostics I [http:192.0.2.0:34692
<admin>@<EXAMPLE>.LOCAL] Broadcasting dump cache value request for key: ExampleClassName
0xE76E32CC3130411F887BD983C63E9E96