Using NexJ CRM system monitoring tools

You can generate a package that contains diagnostic information directly from  NexJ System Admin Console . As you do not need to rely on other teams within your organization to generate this package, you can more quickly provide information to NexJ for troubleshooting purposes.

This functionality is only available when the full JDK is used.

This section provides information about monitoring tools for NexJ Model Server. For information about system dumps, which are specific to WebSphere application server, see Generating a system dump in the system administration documentation for the 8.9 release.

Configuring the default package location

You can configure the default package location for data generated for diagnostic operations associated with system health monitoring.

By default, data is output to following default dump directory: <SERVER_HOME>/data/<node1>/dump

where:

SERVER_HOME

Is the root directory of the NexJ application server instance.

node1

Is the node on which to generate the package.

To configure the default package location:

1. In NexJ System Admin Console, navigate to the Statistics page.
2. In the  Statistics list, expand the node for your NexJ application, then select  Administration . Configuration options for statistics display in the details area.
3. Beside  Diagnostics output directory in the details area, double-click in the Value column. A text entry field displays.
4. Enter an absolute path or a path that is relative to the default data directory. The default package location is updated.

You have configured the default package location. Changes take effect immediately.

Generating dump files

Generating a thread dump

You can generate a thread dump for a node to get a snapshot of the state of all threads.

You can also generate a thread dump for a cluster. Generating a cluster-level thread dump creates a separate thread dump file for each node in the cluster. The name and location of each thread dump file display in the log of the corresponding node. Thread dumps for clusters and for individual nodes other than the current node execute asynchronously.

If your environment file includes the statAvailability attribute, ensure that it is set to distributed to enable thread dumps at the cluster level.

To generate a thread dump for a node:

1. In NexJ System Admin Console, navigate to the Statistics page.
2. In the list in the Statistics area, locate and expand the node on which to generate the thread dump, and select  Diagnostics. For example, navigate to: < application_name>/<node_name>/Diagnostics
Available commands display in the details area on the bottom right.
3. Double-click the  dumpThreads command in the details area. The Invoke dumpThreads command dialog opens.
4. Click  Invoke . The thread dump executes. If the thread dump is for the current node, the name and location of the dump file display in the Result area and the log. If the thread dump is for a different node, the name and location of the dump file display only in the log of the node.
5. Click the  Close button 

   

    on the top right corner of the dialog. The Invoke dumpThreads command dialog closes.

You have generated a thread dump for a node.

Generating a heap dump

You can generate a heap dump for a node to get the state of Java memory.

You can also generate a heap dump for a cluster. Generating a cluster-level heap dump creates a separate heap dump file for each node in the cluster. The name and location of each heap dump file display in the log of the corresponding node.

Heap dumps execute asynchronously, both for clusters and individual nodes. 

If your environment file includes the statAvailability attribute, ensure that it is set to distributed to enable thread dumps at the cluster level.

To generate a heap dump for a node:

1. In NexJ System Admin Console, navigate to the Statistics page.
2. In the list in the Statistics area, locate and expand the node on which to generate the heap dump, and select Diagnostics. For example, navigate to:
   < application_name>/<node_name>/Diagnostics
Available commands display in the details area on the bottom right.
3. Double-click the  dumpHeap  command in the details area. The Invoke dumpHeap command dialog opens. 
4. Click Invoke. The heap dump executes. If the heap dump is for the current node, the location of the dump file displays in the Result area, and the name and location of the dump file display in the log. If the heap dump is for a different node, the name and location of the dump file display in the log of the node.
5. Click the Close button 

   

    on the top right corner of the dialog. The Invoke dumpHeap command dialog closes.

You have generated a heap dump for a node.

Generating a model trace dump

Generate a model trace dump if you want to review the Scheme stack traces across the threads of the JVM.

To generate a model trace dump for a node:

1. In NexJ System Admin Console, navigate to the Statistics page.
2. In the list in the Statistics area, locate and expand the node on which to generate the heap dump, and select Diagnostics. For example, navigate to:
   < application_name>/<node_name>/Diagnostics
Available commands display in the details area on the bottom right.
3. Double-click the dumpModel command in the details area. The Invoke dumpModel command dialog opens. 
4. Click Invoke. The model dump executes. If the model dump is for the current node, the full path and file name relative to the server root displays in the Result area, and is additionally recorded in the log. If the model dump is for a different node, the file path for the dump file is recorded in the log of the node that the dump is for.
5. Click the Close button 

   

    on the top right corner of the dialog. The Invoke dumpModel command dialog closes.

You have generated a model trace dump for a node.

You can also generate a model trace dump for a cluster. Generating a cluster-level dump creates a separate dump file for each node in that cluster. The name and location of each dump file is recorded in the logs of the corresponding nodes. The model trace dump executes asynchronously, both for clusters and individual nodes. 

If your environment file includes the statAvailability attribute, ensure that it is set to distributed to enable dumps at the cluster level.

Collecting application logs in a zipped folder

If using NexJ Server, you can consolidate all application logs for a node into a single zipped folder. You can collect logs of all available events on a node or events that occurred during a specific period.

You can also collect application logs for a cluster. Generating application logs at the cluster level creates a separate zipped folder with the application logs for each node in the cluster. The name and location of the zipped folder display in the log of the corresponding node. Application logs for clusters are collected asynchronously.

To collect application logs for a node:

1. In NexJ System Admin Console, navigate to the Statistics page.
2. In the list in the Statistics area, locate and expand the node for which you want to collect application logs, and select Diagnostics. For example, navigate to:
   < application_name>/<node_name>/Diagnostics
Available commands display in the details area on the bottom right.
3. In the details area, click  collectApplicationLogs . The Invoke collectApplicationLogs command dialog opens.
4. [Optional] If you want to collect logs for a specific period, you can specify the start time and end time.
5. Click Invoke. The application log collection executes and the name and location of the zipped folder display in the Result area and the log. If you specified a date and time range, the application log collection will contain all log files modified during specified period and the earliest file modified after the specified end time.
6. Click the Close button 

   

    on the top right corner to close the dialog.

You have collected application logs for a node in a zipped folder.

Configuring persistent statistics

Configure persistent statistics to store NexJ CRM performance statistics in the StatDatabase data source, in InfluxDB, or both. Persistent statistics are historical performance statistics collected from NexJ CRM that are maintained across server restarts.

To store persistent statistics in InfluxDB only, you must also add the required mixin to the environment file and create an InfluxDB channel connection. For more information, see  Persisting statistics to InfluxDB .

If you want to store persistent statistics in both the StatDatabase data source and InfluxDB, you must set the required environment file property to true. For more information, see Persisting statistics to StatDatabase and InfluxDB .

Persistent statistics are enabled by default in the NexJ model. NexJ System Admin Console configuration overrides the environment file configuration. For more information about configuring persistent statistics in the environment file, see InfluxDB channel connection settings for a typical NexJ CRM deployment.

To configure persistent statistics:

1. In NexJ System Admin Console, navigate to the Statistics page.
2. In the  Statistics list, expand the node for your NexJ application, then select  Administration . Configuration options for statistics display in the details area.
3. To enable or disable persistent statistics:
   1. Beside  Persistent Statistics Enabled  in the details area, double-click in the Value column. A text entry field displays.
   2. To enable persistent statistics, enter Y. To disable persistent statistics, enter N. The persistent statistics are enabled or disabled.
4. [Optional] To specify the amount of time that persisted statistics are stored before being deleted, do one of the following:
   1. If storing the statistics in the StatDatabase, enter a timeout value (in minutes) in the Value column beside  Persistent Statistics Timeout (min) in the details area. The default value is 10,080 (7 days).
   2. If storing the statistics in InfluxDB, enter the timeout value in InfluxDB. For instructions, see the InfluxDB documentation at https://docs.influxdata.com/influxdb. The persistent statistics timeout value is updated.

5. [Optional] To specify the frequency at which to sample statistics, do one of the following:
   

   

   Sampling intervals smaller than the default setting might affect application performance. Small sampling intervals can also create fluctuations in sample data that can make analysis difficult.

   1. If storing the statistics in the StatDatabase, enter a timeout value (in seconds) in the Value column beside  Statistics Sampling Interval (s)  in the details area. The default value is 60 (1 minute).
   2. If storing the statistics in InfluxDB, enter the timeout value in InfluxDB. For instructions, see the InfluxDB documentation at https://docs.influxdata.com/influxdb. 
      The sampling interval value is updated.
      

Persistent statistics are configured for NexJ CRM. Changes take effect immediately.

Persisting statistics to InfluxDB

You can persist performance statistics to InfluxDB to leverage the real-time analytics and data visualization capabilities of InfluxDB for your NexJ application.

InfluxDB is a time series database that enables scalable storage, retrieval, analytics, and presentation of large volumes of data. InfluxDB works with data visualization tools such as Grafana or Chronograf to present performance statistics in easy-to-interpret formats such as graphs and dashboards.

You can only store performance statistics in InfluxDB. Storage of client performance metrics and integration logs is currently not supported.

The  dumpStats  diagnostic command in  NexJ System Admin Console  only exports statistics persisted to the StatDatabase data source. Statistics persisted to InfluxDB cannot be exported using this command.

To persist performance statistics to InfluxDB:

1. Ensure persistent statistics are enabled in the Statistics page of NexJ System Admin Console. For more information, see Configuring persistent statistics .

2. Optional] You can use InfluxDB to specify the amount of time that persisted statistics are stored before being deleted.
   

   

   For information about how to specify the persistent statistics timeout value, see the InfluxDB documentation at https://docs.influxdata.com/influxdb.

3. Create a monitor:influxdb:HTTPSender channel connection. When configuring the channel connection, enter <influxdb host>:<port>/write?db=<database> in the  URL field.
   

   

   For instructions on creating and configuring channel connections, see Creating channel connections. For detailed information about InfluxDB channel connection settings, see Channel connection settings. 

Persistent statistics are configured for storage in InfluxDB. If your NexJ application is currently running, you must redeploy it for the changes to take effect.

Persisting statistics to StatDatabase and InfluxDB

If desired, you can persist performance statistics to the StatDatabase data source and InfluxDB simultaneously using the sysStat.multiplePersist property in your environment file.

To persist statistics to the StatDatabase data source and InfluxDB:

1. Enable persistent statistics. For more information, see Configuring persistent statistics.
2. Add the monitoring and alerting mixin to the environment. For more information, see Persisting statistics to InfluxDB.
3. Add the sysStat.multiplePersist property the root tag of your environment file with the value set to true, in the following format:
   <Environment... sysStat.multiplePersist="true" ...>


Persistent statistics are configured for storage in both the StatDatabase data source and InfluxDB. If your NexJ application is currently running, you must redeploy it for the changes to take effect.

Exporting persistent statistics

You can export persistent statistics for a node to a CSV file. The CSV file includes a value, a timestamp, an MBean path, and a node name for each attribute included in the statistics.

You can also export persistent statistics for a cluster. Carrying out a cluster-level export creates a separate zipped folder with the persistent statistics for each node in the cluster. The name and location of the zipped folder display in the log of the corresponding node.

This feature currently exports statistics persisted to the StatDatabase only. Statistics persisted to InfluxDB cannot be exported using this feature.

Persistent statistics are exported asynchronously, both for clusters and individual nodes.

To export persistent statistics for a node:

1. In NexJ System Admin Console, navigate to the Statistics page.
2. In the list in the Statistics area, locate and expand the node for which you want to collect application logs, and select  Diagnostics. For example, navigate to:
   < application_name>/<node_name>/Diagnostic
Available commands display in the details area on the bottom right.
3. Double-click the  dumpStats command in the details area.
   The Invoke dumpStats command dialog opens.
4. Enter the period for which you want to export statistics.
5. [Optional] To restrict the output to specific statistics, you can enter a Scheme WHERE clause containing the attributes or values you want to export. For example, if you only want to view the total message count, enter the following expression in  Scheme clause: 
   (= attribute "Total message count")
6. Click  Invoke . 
   The persistent statistics are exported. If you are exporting statistics for the current node, the name and location of the dump file display in the Result area and the log. If you are exporting statistics for a different node, the name and location of the dump file display only in the log of the node.
7. Click the  Close button 

   

    on the top right corner to close the Invoke dumpStats command dialog.

You have exported persistent statistics for a node.