NexJ Admin Console and NexJ System Admin Console provide tools to configure and monitor your NexJ application, including NexJ Model Server.

This section covers those features that are specific to Model Server.  For more general information on monitoring your application, including capturing logs, thread dumps and heap dumps, see Monitoring NexJ applications.

Logging activity

All NexJ applications running on Model Server use Apache Log4j, a Java-based logging utility, to generate log files.

By default, Log4j uses a configuration file called log4j.properties. It is located in the <SERVER_HOME>\conf directory. To customize logging functionality for your specific business needs, such as logging information for specific components, individual users, or both, you can either customize the file, or use the Run-time logging feature available in NexJ System Admin Console. The System Admin Console is the recommended method, and has the advantage that levels are applied for all nodes in the cluster.

Changes made to the configuration file will take effect every 60 seconds.

Log files are output to a directory specified in the log4j.properties file, and can be opened using a text editor. By default, log files output to <SERVER_HOME>\log\<nodename>.log, where <nodename> is the name of the node. A separate log file is output for each node in a cluster.

You can also use third-party tools, such as Splunk, to generate, manage, and search application logs. For detailed information about managing your logs using specific third-party tools, refer to the documentation for the tool you want to use.

Changing the log4j.properties file location

An alternate configuration file can be specified by passing log4j.configurationFile as a JVM argument when starting the server. For example:
modeld.bat "-Dlog4j.configurationFile=C:/log4jdebug.properties"

In this case, log4jdebug.properties will be used to configure Log4j.

Updating logging levels

Model Server supports run-time logging changes when it is run in standalone mode.

When configuration changes are made, the updates are applied incrementally instead of resetting all of the log settings. This means that if a log category is set to DEBUG and you want to set it back to INFO, it must be set explicitly to INFO. Commenting it out or removing it from the configuration file will keep the category at the DEBUG level.

For more information about this functionality, see the Apache log4j website.

Configuring component logging and named user logging

You can enable component type logging and named user logging by specifying properties in the log4j.properties file.

Component logging enables you to log specific components of the model for troubleshooting purposes. For example, you can enable logging for a class in the business model layer, such as the User class.

Named user logging enables you to target specific users for detailed logging without changing the general logging level for the rest of the user base. Filtering log information for individual users helps troubleshoot issues in a production environment with minimal performance overhead.

Configuring component logging

Component logging enables logging of specific components of the model for troubleshooting purposes. For example, you can enable logging for a class in the business model layer, such as the User class.

Define which component to log using the following syntax:

<unique name>.name = nexj.model.<componenttype>.<identifier>
<unique name>.level = <level>

Where <componenttype> indicates the component type and <identifier> refers to the associated filename.

For example, to enable logging for the user class, add the following to the log4j.properties file:

nexj-model-class-User.name = nexj.model.class.User
nexj-model-class-User.level = ALL

Available component types

The following table lists the component types that are available for logging.

Configuring named user logging

Enable and configure named user logging by specifying user and class properties in the log4j.properties file.

Named user logging is configured by adding a properties to enable named user logging for a specified user, followed by one or more property pairs specifying the user and the components you want to log for the user. If the log level set for a user (for example, DEBUG) is different from the log level set for a component associated with the user (for example, ALL), the output will be based on the higher of the log levels (which, in this case, is DEBUG).

There is no restriction on the number of users you can target. Also, each user may have their own set of specific classes that are logged at different levels. 

Named user logging highlights user actions in application log files. By default, application logs do not contain a user reference:

2015-12-11 15:44:33,134 DUMP [nexj.core.rpc.http.session.SessionManager]
 (http-127.0.0.1-8080-6:[SessionId=C4616C6EE08438F96145E1A9B42B0F28.node1]) Started
 intercept

Whereas named user logging prints the name of the user in the details:

2015-12-11 15:44:33,134 DUMP
 [nexj.user.username.core.rpc.http.session.SessionManager] (http-127.0.0.1-8080-6:
[SessionId=C4616C6EE08438F96145E1A9B42B0F28.node1]) Started intercept

The following properties and parameters enable and control named user logging. You add them to the log4j.properties file to highlight user interactions with the application in the log files.

Enable logging for a targeted user

The following properties enables named user logging for a targeted user:

logger.nexj-enable-user-<username>.name=nexj.enable.user.<username>
logger.nexj-enable-user-<username>.level=ALL

Where <username> specifies the name of the user to enable logging for.

To disable named user logging for the specified user without restarting the server, set the logging level to OFF.
logger.nexj-enable-user-<username>.level=OFF

Specify logging level for a targeted user

Once named user logging has been enabled for a targeted user, you can specify which classes should be logged and at what logging level.

To set the logging level for all classes, use:

logger.nexj-user-<username>.name=nexj.user.<username>
logger.nexj-user-<username>.level=<logging_level>

Where:

• <username> specifies the name of the user to define logging levels for.
• <logging_level> specifies the logging level for a class or group of classes. The following levels of logging detail are available in log4j: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, and ALL.

To set the logging level for a specific class or group of classes, use:

logger.nexj-user-<username>-<partial-class>.name=nexj.user.<username>.<partial_class>
logger.nexj-user-<username>-<partial-class>.level=<logging_level>

Where <partial_class> is the suffix of the class name after excluding the package root.

Example

The following example defines separate logging for three different users, at various levels of logging detail.

• For the first user, Joseph Sanders, Log4j will log the activity for the nexj.model.class.User class at the ALL level. All other activities will be logged at the DEBUG level.

  logger.nexj-enable-user-jsanders.name=nexj.enable.user.jsanders
  logger.nexj-enable-user-jsanders.level=ALL
  
  logger.nexj-user-jsanders-model-class-User.name=nexj.user.jsanders.model.class.User
  logger.nexj-user-jsanders-model-class-User.level=ALL
  
  logger.nexj-user-jsanders.name=nexj.user.jsanders
  logger.nexj-user-jsanders.level=DEBUG

• For the second user, Wendy Johnson, all activity for all classes will be logged at the DEBUG level.

  logger.nexj-enable-user-wjohnson.name=nexj.enable.user.wjohnson
  logger.nexj-enable-user-wjohnson.level=ALL
  
  logger.nexj-user-wjohnson.name=nexj.user.wjohnson
  logger.nexj-user-wjohnson.level=DEBUG

• For the third user, Tim Smith, Log4j logs all activity for the nexj.core.rpc.http.session.SessionManager class, and all the nexj.core.persistence classes at the ALL level. The logging level for all other activity will be inherited from the server defaults.

  logger.nexj-enable-user-tsmith.name=nexj.enable.user.tsmith
  logger.nexj-enable-user-tsmith.level=ALL
  
  logger.nexj-user-tsmith-core-rpc-http-session-SessionManager.name=nexj.user.tsmith.core.rpc.http.session.SessionManager
  logger.nexj-user-tsmith-core-rpc-http-session-SessionManager.level=ALL
  
  logger.nexj-user-tsmith-core-persistence.name=nexj.user.tsmith.core.persistence
  logger.nexj-user-tsmith-core-persistence.level=ALL

Generating advanced model server logs

Output from the standard model server logging can sometimes be insufficient for debug. You can use the following options for more advanced control over the behavior of the logger.

Filtering with categories

Log statements are categorized by source. All model server categories begin with nexj.core. Some useful categories are:

• integration
• meta
• monitoring
• persistence
• rpc
• runtime
• scripting

Categories allow you to set different log options for different loggers. For example, if you only want to log output from the persistence layer at DEBUG level, set the console to INFO level and add the following statements to the log4j.properties:

logger.nexj-user-nexj-core-persistence.name=nexj.core.persistence
logger.nexj-user-nexj-core-persistence.level=DEBUG

This will enable all log statements at INFO or above but will enable nexj.core.persistence.* log statements at DEBUG or above.  

To set more specific logging for database persistence, change the statements to the following:

logger.nexj-user-nexj-core-persistence-sql.name=nexj.core.persistence.sql
logger.nexj-user-nexj-core-persistence-sql.level=DEBUG

You may specify the category at any level, starting from “nexj” down to a specific class, for example, nexj.core.persistence.sql.SQLAdapter.

Filtering log output

There are instances where the specified loggers output is too verbose resulting in possible performance degradation and excessive log file rollover. It may be beneficial to apply a filter such that the log output is constrained to better isolate the target output and reduce churn.The filter statements below restricts the corresponding named user logger output to the time interval from 8am to 9am.

logger.nexj-enable-user-wm1.name=nexj.enable.user.wm1
logger.nexj-enable-user-wm1.level=ALL
logger.nexj-user-wm1.name=nexj.user.wm1
logger.nexj-user-wm1.level=DEBUG

logger.nexj-user-wm1.filter.time.type=TimeFilter
logger.nexj-user-wm1.filter.time.start=08:00:00
logger.nexj-user-wm1.filter.time.end=09:00:00
logger.nexj-user-wm1.filter.time.onMatch=ACCEPT
logger.nexj-user-wm1.filter.time.onMismatch=DENY

You can find additional details on Log4j 2 filtering capabilities at https://logging.apache.org/log4j/2.x/manual/filters.html.

Finding Exchange synchronization logs

If your administrator has enabled Exchange application logging, Exchange synchronization process logs in the database are included in your application logs. This allows you to find errors and warnings associated with Exchange synchronization operations using your application logs. 

You can find Exchange synchronization events by searching for ExchangeFolder and ExchangeProcess entries in the log.

The following snippet shows Exchange synchronization entries from an example application log.

...
; 14:44:09,211 WARN  [ExchangeProcess] Autodiscover attempt for email michael@jmail.com failed.
; 14:44:09,231 ERROR [ExchangeProcess] Unable to create links for user janet@jmail.com on Exchange server #1.
...
; 14:45:34,296 INFO  [ExchangeFolder] Inbound sync enabled for "michael@jmail.com/Tasks".
; 14:45:34,296 INFO  [ExchangeFolder] Outbound sync enabled for "michael@jmail.com/Tasks".
; 14:45:34,296 INFO  [ExchangeFolder] Inbound sync enabled for "michael@jmail.com/Contacts".
; 14:45:34,297 INFO  [ExchangeFolder] Outbound sync enabled for "michael@jmail.com/Contacts".
; 14:45:54,227 INFO  [ExchangeFolder] Inbound sync disabled for "michael@jmail.com/Contacts".
; 14:45:54,228 INFO  [ExchangeFolder] Outbound sync enabled for "michael@jmail.com/Contacts".
...
; 14:47:47,984 INFO  [ExchangeFolder] Inbound sync enabled for "janet@jmail.com/Tasks".
; 14:45:34,296 INFO  [ExchangeFolder] Outbound sync enabled for "janet@jmail.com/Tasks".
; 14:45:34,296 INFO  [ExchangeFolder] Inbound sync enabled for "janet@jmail.com/Contacts".
; 14:45:34,297 INFO  [ExchangeFolder] Outbound sync enabled for "janet@jmail.com/Contacts".
; 14:45:54,227 INFO  [ExchangeFolder] Inbound sync disabled for "janet@jmail.com/Contacts".
; 14:45:54,228 INFO  [ExchangeFolder] Outbound sync enabled for "janet@jmail.com/Contacts".

Configuring exception logging in Model Server environments

By default, all exceptions are logged and include JVM stack traces and, where applicable, include the NexJ model stack trace.  An example is shown below:

Unknown formula validate error: java.lang.ArithmeticException: / by zero
 at nexj.core.meta.Primitive.divide(Primitive.java:5692)
 at nexj.core.meta.Primitive$46.invoke(Primitive.java:631)
 at nexj.core.meta.Primitive.divide(Primitive.java:5812)
 at Formula.validate(text)$main(class:Formula.validate(text)$main:17)
 at Formula.validate(text)$main(class:Formula.validate(text)$main:4)
 at Formula.testFormulaValidationWithFields(unittest:Formula.testFormulaValidationWithFields:68)
 at ---.---(---:0)
 at nexj.core.scripting.Machine.invoke(Machine.java:748)
 at nexj.core.testing.unit.UnitTestPlayer.run(UnitTestPlayer.java:695)
 at nexj.core.testing.unit.UnitTestPlayer.run(UnitTestPlayer.java:500)
 at nexj.core.tools.UnitTestTool.runTests(UnitTestTool.java:507)
 at nexj.core.tools.UnitTestTool.end(UnitTestTool.java:569)
 at nexj.core.tools.GenericTool.run(GenericTool.java:131)
 at nexj.core.tools.UnitTestTool.main(UnitTestTool.java:826)

The ---.---(—:0) stack trace element acts as a separator between the Java stack traces (below) and NexJ stack traces (above) so that you can easily find the NexJ model stack trace. You can configure this behavior by using the following log categories:

• Suppresses Java portion:

  • logger.nexj-core-scripting-Machine-java.name=nexj.core.scripting.Machine.java

  • logger.nexj-core-scripting-Machine-java.level=OFF

• Suppresses NexJ model portion:

  • logger.nexj-core-scripting-Machine-scm.name=nexj.core.scripting.Machine.scm

  • logger.nexj-core-scripting-Machine-scm.level=OFF

If both are disabled then the Java portion is still included.

Error and exception entries in log files

When troubleshooting issues with your NexJ application, you may want to specifically review the ERROR and EXCEPTION entries in log files. 

Example error message

The following are examples of error messages in Model Server application logs.

2008-10-23 15:14:00,723 ERROR [nexj.core.rpc.jms.ra.JMSConsumerPool] 
(WorkManager(3)-6:)
Error in JMSConsumerPool@20219443
 (tx=true, config=JMSConsumerConfig@1549250874
      (cf=XAConnectionFactory,
       destination=queue/nexj/NEXJ_QUEUE,
       broadcast=false,
       durable=false,
       maxPoolSize=4,
       errorCF=XAConnectionFactory,
       errorQueue=queue/nexj/NEXJ_ERROR,
       errorCount=3
)

Example exception message

The following is an example of an exception message in Model Server application logs.

javax.jms.JMSSecurityException: User: null is NOT authenticated
   at org.jboss.mq.security.SecurityManager...
   at ...

Accessing JMX statistics remotely

You can access JMX statistics remotely using any of the following methods:

• By adding JVM arguments that expose JMX statistics through Java Remote Method Invocation (RMI). This enables you to monitor and manage the statistics using the using the built-in management utilities in JVMs. For more information, see https://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html. For information about adding JVM arguments, see Specifying JVM arguments and system properties.
• By persisting your performance statistics to InfluxDB and then exporting JMX statistics from InfluxDB. This enables you to view the data through visualization tools such as Grafana. For more information about persisting statistics to InfluxDB, see Persisting statistics to InfluxDB.

• Through HTTP, using the nexj-jmxclient.jar file. This enables you to view JMX statistics without using JVM arguments and based on your NexJ application privileges in NexJ System Admin Console. For more information, see Viewing JMX statistics remotely.

  The JMXManage privilege allows the user to view and edit JMX statistics in NexJ System Admin Console.

Monitoring Model Server using Grafana dashboards

You can monitor the performance of Model Server at a glance with Grafana dashboards. For more information, see Monitoring NexJ Model Server with Grafana dashboards.

Viewing Model Server properties in NexJ System Admin Console

You can view Model Server properties by expanding the nexj.teee node on the Statistics page in NexJ System Admin Console. The properties displayed here include the Tomcat connector properties for your configuration.

Refer to the Apache Tomcat documentation for a description of all configurable worker properties: https://tomcat.apache.org/tomcat-8.5-doc/config/http.html.

Run-time logging in NexJ System Admin Console

You can use run-time logging to view and update logger levels at run time. Use NexJ System Admin Console to configure loggers with their targeted levels, and apply the configuration across the nodes in the cluster.

Loggers in the configuration are separated by semi-colons; for example:
com.company.package.subpackage.class_a=DEBUG; com.company.package.subpackage_b.class_b=AL

nexj-user-nexj-core-persistence-sql.name=nexj.core.persistence.sql
nexj-user-nexj-core-persistence-sql.level=DEBUG

nexj.core.persistence.sql=DEBUG

To edit the configuration in NexJ System Admin Console, select the Logging tab, click the edit button , and provide the required target log levels. Any level supported by the configured logging platform is valid. For Model Server, Apache Log4j is the default platform, which supports: ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF.

After you have made your changes, and to apply the configuration, click Apply.

You can remove a logger from the configuration to reset its level to that which was configured when the application started, or if no such level was configured, to inherit from ancestry.

In the Nodes section, click a node to view its details in the Log Level Details section. The synchronization status of each node indicates if the node’s current logger levels are consistent with the configuration. If all levels match, the node is indicated with "Synchronized," and "Unsychronized" otherwise. A node that is not available is indicated as "Inactive."

The Log Level Details section provides the following information:

• It lists the loggers that have their current level set explicitly (not inherited from ancestry), as well as the loggers in the configuration.
• It displays each logger's effective level (the root's level if no ancestor is set explicitly), and the level that will be applied (in brackets), if the level is not visible in the target configuration (because it has been removed).
• If the target and current levels vary, it shows an asterisk next to a logger's current level.
• If the logger is not known to the logging platform, it shows <unspecified> for a logger's level.

The details are available per node.

After applying the configuration, the Log Level Details can be refreshed by clicking the Refresh button .