Monitoring Model Server

NexJ Admin Console and NexJ System Admin Console provide tools to configure and monitor your NexJ application, including NexJ Model Server.

This section covers those features that are specific to Model Server.  For more general information on monitoring your application, including capturing logs, thread dumps and heap dumps, see Monitoring NexJ applications.

Logging activity

All NexJ applications running on Model Server use Apache Log4j, a Java-based logging utility, to generate log files.

By default, Log4j uses a configuration file called log4j.properties. It is located in the <SERVER_HOME>\conf directory. To customize logging functionality for your specific business needs, such as logging information for specific components, individual users, or both you can either customize the file or use the Run-time logging feature available in NexJ System Admin Console. The System Admin Console is the recommended method and has the advantage that levels are applied for all nodes in the cluster.

Changes made to the configuration file will take effect every 60 seconds.

Log files are output to a directory specified in the log4j.properties file and can be opened using a text editor. By default, log files output to <SERVER_HOME>\log\<nodename>.log, where <nodename> is the name of the node. A separate log file is output for each node in a cluster.

You can also use third-party tools, such as Splunk, to generate, manage, and search application logs. For detailed information about managing your logs using specific third-party tools, refer to the documentation for the tool you want to use.

Changing the log4j.properties file location

An alternate configuration file can be specified by passing log4j.configuration as a JVM argument when starting the server. For example:
modeld.bat "-Dlog4j.configuration=C:/log4jdebug.properties"

In this case, log4jdebug.properties will be used to configure Log4j.

Updating logging levels

Model Server supports runtime logging changes when it is run in standalone mode.

When configuration changes are made, the updates are applied incrementally instead of resetting all of the log settings. This means that if a log category is set to DEBUG and you want to set it back to INFO, it must be set explicitly to INFO. Commenting it out or removing it from the configuration file will keep the category at a DEBUG level.

For more information about this functionality, see the Apache log4j website.

Configuring component logging and named user logging

You can enable component type logging and named user logging by specifying properties in the log4j.properties file.

Component logging enables you to log specific components of the model for troubleshooting purposes. For example, you can enable logging for a class in the business model layer, such as the User class.

Named user logging enables you to target specific users for detailed logging without changing the general logging level for the rest of the user base. Filtering log information for individual users helps troubleshoot issues in a production environment with minimal performance overhead.

Configuring component logging

Component logging enables logging of specific components of the model for troubleshooting purposes. For example, you can enable logging for a class in the business model layer, such as the user class.

You define which component to log using the following syntax:
nexj.model.<componenttype>.<identifier>

Where <componenttype> indicates the component type and <identifier> refers to the associated filename.

For example, to enable logging for the user class, add the following to the log4j.properties file:
nexj.model.class.User=ALL

Available component types

The following table lists the component types that are available for logging.

For the SOA implementation component, replace delimiter colons (:) with category delimiter periods (.). If the component includes a version number, exclude the version number. For example, if the service name for the SOA implementation Audit is nexj:soa:Audit:1.0, use the identifier nexj.soa.Audit.

Configuring named user logging

Enable and configure named user logging by specifying user and class properties in the log4j.properties file.

Named user logging is configured by adding a property to enable named user logging for a specified user, followed by one or more properties specifying the user and the components you want to log for the user. If the log level set for a user (for example, DEBUG) is different from the log level set for a component associated with the user (for example, ALL), the output will be based on the higher of the log levels (which, in this case, is DEBUG).

There is no restriction on the number of users you can target. Also, each user may have their own set of specific classes that are logged at different levels. 

Named user logging highlights user actions in application log files. By default, application logs do not contain a user reference:

2015-12-11 15:44:33,134 DUMP [nexj.core.rpc.http.session.SessionManager]
 (http-127.0.0.1-8080-6:[SessionId=C4616C6EE08438F96145E1A9B42B0F28.node1]) Started
 intercept

Whereas named user logging prints the name of the user in the details:

2015-12-11 15:44:33,134 DUMP
 [nexj.user.username.core.rpc.http.session.SessionManager] (http-127.0.0.1-8080-6:
[SessionId=C4616C6EE08438F96145E1A9B42B0F28.node1]) Started intercept

The following properties and parameters enable and control named user logging. You add them to the log4j.properties file to highlight user interactions with the application in the log files.

Enable logging for a targeted user

The following property enables named user logging for a targeted user:
log4j.logger.nexj.enable.user.<username>=ALL

Where <username> specifies the name of the user to enable logging for.

To disable named user logging for the specified user without restarting the server, set the logging level to OFF.
log4j.logger.nexj.enable.user.<username>=OFF

Specify logging level for a targeted user

Once named user logging has been enabled for a targeted user, you can specify which classes should be logged and at what logging level.

To set the logging level for all classes:
log4j.logger.nexj.user.<username>=<logging_level>

Where:

• <username> specifies the name of the user to define logging levels for.
• <logging_level> specifies the logging level for a class or group of classes. The following levels of logging detail are available in log4j: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, and ALL.

To set the logging level for a specific class or group of classes:
log4j.logger.nexj.user.<username>.<partial_class>=<logging_level>

Where <partial_class> is the suffix of the class name after excluding the package root.

Example

The following example defines separate logging for three different users, at various levels of logging detail.

• For the first user, Joseph Sanders, log4j will log the activity for the nexj.model.class.User class at the ALL level. All other activities will be logged at the DEBUG level.

  TEXT

  log4j.logger.nexj.enable.user.jsanders=ALL
  log4j.logger.nexj.user.jsanders.model.class.User=ALL
  log4j.logger.nexj.user.jsanders=DEBUG

• For the second user, Wendy Johnson, all activity for all classes will be logged at the DEBUG level.

  TEXT

  log4j.logger.nexj.enable.user.wjohnson=ALL
  log4j.logger.nexj.user.wjohnson=DEBUG

• For the third user, Tim Smith, log4j logs all activity for the nexj.core.rpc.http.session.SessionManager class and all the nexj.core.persistence classes at the ALL level. The logging level for all other activity will be inherited from the server defaults.

  TEXT

  log4j.logger.nexj.enable.user.tsmith=ALL
  log4j.logger.nexj.user.tsmith.core.rpc.http.session.SessionManager=ALL
  log4j.logger.nexj.user.tsmith.core.persistence=ALL

Generating advanced model server logs

Output from the standard model server logging can sometimes be insufficient for debug. You can use the following options for more advanced control over the behavior of the logger.

Filtering with Categories

Log statements are categorized by source. All model server categories begin with nexj.core.  Some useful categories are:

• integration
• meta
• monitoring
• persistence
• rpc
• runtime
• scripting

Categories allow you to set different log options for different loggers. For example, if you only want to log output from the persistence layer at DEBUG level, set the console to INFO level and add the following statement to the VM section of the Scheme Console settings:

-Dlog4j.category.nexj.core.persistence=DEBUG

This will enable all log statements at INFO or above but will enable nexj.core.persistence.* log statements at DEBUG or above.  

To set more specific logging for database persistence, change the statement to the following:

-Dlog4j.category.nexj.core.persistence.sql=DEBUG

To see more of the available categories, set your layout to the following and then run the server console:

-Dlog4j.appender.console.layout.ConversionPattern="; [%c]%m%n"

You may specify the category at any level, starting from “nexj” down to a specific class, for example, nexj.core.persistence.sql.SQLAdapter.

Logging Specific Metadata

You can control the the breadth and scope of the information output to the log by specifying logging as desired down to the metadata level. This applies to classes, integration services, and business model workflows. For more information, see Configuring component logging.

Finding Exchange synchronization logs

If your administrator has enabled Exchange application logging, Exchange synchronization process logs in the database are included in your application logs. This allows you to find errors and warnings associated with Exchange synchronization operations using your application logs. 

Exchange application logging is enabled through the Exchange App Logs global application setting. For more information, see Enabling application features in the system administration information.

You can find Exchange synchronization events by searching for ExchangeFolder and ExchangeProcess entries in the log.

The following snippet shows Exchange synchronization entries from an example application log.

...
; 14:44:09,211 WARN  [ExchangeProcess] Autodiscover attempt for email michael@jmail.com failed.
; 14:44:09,231 ERROR [ExchangeProcess] Unable to create links for user janet@jmail.com on Exchange server #1.
...
; 14:45:34,296 INFO  [ExchangeFolder] Inbound sync enabled for "michael@jmail.com/Tasks".
; 14:45:34,296 INFO  [ExchangeFolder] Outbound sync enabled for "michael@jmail.com/Tasks".
; 14:45:34,296 INFO  [ExchangeFolder] Inbound sync enabled for "michael@jmail.com/Contacts".
; 14:45:34,297 INFO  [ExchangeFolder] Outbound sync enabled for "michael@jmail.com/Contacts".
; 14:45:54,227 INFO  [ExchangeFolder] Inbound sync disabled for "michael@jmail.com/Contacts".
; 14:45:54,228 INFO  [ExchangeFolder] Outbound sync enabled for "michael@jmail.com/Contacts".
...
; 14:47:47,984 INFO  [ExchangeFolder] Inbound sync enabled for "janet@jmail.com/Tasks".
; 14:45:34,296 INFO  [ExchangeFolder] Outbound sync enabled for "janet@jmail.com/Tasks".
; 14:45:34,296 INFO  [ExchangeFolder] Inbound sync enabled for "janet@jmail.com/Contacts".
; 14:45:34,297 INFO  [ExchangeFolder] Outbound sync enabled for "janet@jmail.com/Contacts".
; 14:45:54,227 INFO  [ExchangeFolder] Inbound sync disabled for "janet@jmail.com/Contacts".
; 14:45:54,228 INFO  [ExchangeFolder] Outbound sync enabled for "janet@jmail.com/Contacts".

Configuring exception logging in Model Server environments

By default, all exceptions are logged and include JVM stack traces and, where applicable, include the NexJ model stack trace.  An example is shown below:

Unknown formula validate error: java.lang.ArithmeticException: / by zero
 at nexj.core.meta.Primitive.divide(Primitive.java:5692)
 at nexj.core.meta.Primitive$46.invoke(Primitive.java:631)
 at nexj.core.meta.Primitive.divide(Primitive.java:5812)
 at Formula.validate(text)$main(class:Formula.validate(text)$main:17)
 at Formula.validate(text)$main(class:Formula.validate(text)$main:4)
 at Formula.testFormulaValidationWithFields(unittest:Formula.testFormulaValidationWithFields:68)
 at ---.---(---:0)
 at nexj.core.scripting.Machine.invoke(Machine.java:748)
 at nexj.core.testing.unit.UnitTestPlayer.run(UnitTestPlayer.java:695)
 at nexj.core.testing.unit.UnitTestPlayer.run(UnitTestPlayer.java:500)
 at nexj.core.tools.UnitTestTool.runTests(UnitTestTool.java:507)
 at nexj.core.tools.UnitTestTool.end(UnitTestTool.java:569)
 at nexj.core.tools.GenericTool.run(GenericTool.java:131)
 at nexj.core.tools.UnitTestTool.main(UnitTestTool.java:826)

The ---.---(—:0) stack trace element acts as a separator between the Java stack traces (below) and NexJ stack traces (above) so that you can easily find the NexJ model stack trace. You can configure this behavior by using the following log categories:

• nexj.core.scripting.Machine.java=OFF (suppresses Java portion)
• nexj.core.scripting.Machine.scm=OFF (suppresses NexJ model portion)

If both are disabled then the Java portion is still included.

Error and exception entries in log files

When troubleshooting issues with your NexJ application, you may want to specifically review the ERROR and EXCEPTION entries in log files. 

Example error message

The following are examples of error messages in Model Server application logs.

2008-10-23 15:14:00,723 ERROR [nexj.core.rpc.jms.ra.JMSConsumerPool] 
(WorkManager(3)-6:)
Error in JMSConsumerPool@20219443
 (tx=true, config=JMSConsumerConfig@1549250874
      (cf=XAConnectionFactory,
       destination=queue/nexj/NEXJ_QUEUE,
       broadcast=false,
       durable=false,
       maxPoolSize=4,
       errorCF=XAConnectionFactory,
       errorQueue=queue/nexj/NEXJ_ERROR,
       errorCount=3
)

Example exception message

The following is an example of an exception message in Model Server application logs.

javax.jms.JMSSecurityException: User: null is NOT authenticated
   at org.jboss.mq.security.SecurityManager...
   at ...

Accessing JMX statistics remotely

You can access JMX statistics remotely using any of the following methods:

• By adding JVM arguments that expose JMX statistics through Java Remote Method Invocation (RMI). This enables you to monitor and manage the statistics using the using the built-in management utilities in JVMs. For more information, see https://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html. For information about adding JVM arguments, see Specifying JVM arguments and system properties.
• By persisting your performance statistics to InfluxDB and then exporting JMX statistics from InfluxDB. This enables you to view the data through visualization tools such as Grafana. For more information about persisting statistics to InfluxDB, see Persisting statistics to InfluxDB.

• Through HTTP, using the nexj-jmxclient.jar file. This enables you to view JMX statistics without using JVM arguments and based on your NexJ application privileges, in NexJ System Admin Console. For more information, see Viewing JMX statistics remotely.

  

  The JMXManage privilege allows the user to view and edit JMX statistics in NexJ System Admin Console.

Monitoring Model Server using Grafana dashboards

You can monitor the performance of Model Server at a glance with Grafana dashboards. For more information, see Monitoring NexJ Model Server with Grafana dashboards.

Viewing Model Server properties in NexJ System Admin Console

You can view Model Server properties by expanding the nexj.teee node on the Statistics page in NexJ System Admin Console. The properties displayed here include the Tomcat connector properties for your configuration.

Refer to the Apache Tomcat documentation for a description of all configurable worker properties: https://tomcat.apache.org/tomcat-8.5-doc/config/http.html.

Run-time logging in NexJ System Admin Console

You can use run-time logging to view and update logger levels at run time. Use NexJ System Admin Console to configure loggers with their targeted levels and apply the configuration across the nodes in the cluster.

Loggers in the configuration are separated by semi-colons; for example:
com.company.package.subpackage.class_a=DEBUG; com.company.package.subpackage_b.class_b=AL

The "log4j.logger" prefix that is commonly used in the log4j.properties file must be excluded when specifying logger levels using NexJ System Admin Console.

To edit the configuration in NexJ System Admin Console, select the Logging tab, click the edit button

After you have made your changes, and to apply the configuration, click Apply.

You can remove a logger from the configuration to reset its level to that which was configured when the application started, or if no such level was configured, to inherit from ancestry.

In the Nodes section, click a node to view its details in the Log Level Details section. The synchronization status of each node indicates if the node’s current logger levels are consistent with the configuration. If all levels match, the node is indicated with "Synchronized," and "Unsychronized" otherwise. A node that is not available is indicated as "Inactive."

The Log Level Details section provides the following information:

• It lists the loggers that have their current level set explicitly (not inherited from ancestry), as well as the loggers in the configuration.
• It displays each logger's effective level (the root's level if no ancestor is set explicitly), and the level that will be applied (in brackets), if the level is not visible in the target configuration (because it has been removed).
• If the target and current levels vary, it shows an asterisk next to a logger's current level.
• If the logger is not known to the logging platform, it shows <unspecified> for a logger's level.

The details are available per node.

After applying the configuration, the Log Level Details can be refreshed by clicking the Refresh button