Application monitoring tools have for a long time been able to fetch data continuously over the network using JMX. For example, the CPU load can be obtained from the OperatingSystemMXBean and visualized in JDK Mission Control. JFR provides richer data that is structured, for example stack traces and timestamped values, but until JDK 16 there hasn’t been a way to transfer this information over the network as it occurs.
In JDK 14, API support was added to stream events, illustrated by the following code snippets:
1. Passive, in process:
try (EventStream stream = EventStream.openRepository()) {
stream.onEvent("jdk.JavaMonitorEnter", System.out::println),
stream.start();
}
2. Passive, out of process:
Path path = Path.of("/repository/2021_05_16_09_48_31_60185");
try (EventStream stream = EventStream.openRepository(path) {
stream.onEvent("jdk.JavaMonitorEnter", System.out::println),
stream.start();
}
3. Active, in process:
try (RecordingStream stream = new RecordingStream()) {
stream.enable("jdk.JavaMonitorEnter").withStackTrace();
stream.onEvent("jdk.JavaMonitorEnter", System.out::println),
stream.start();
}
Active here means that the recording lifecycle and event configuration can be controlled by the stream, as seen in the third code snippet with the enabled method. The above APIs work for many scenarios, but not when there is a need to:
- monitor a Java process on a remote host.
- control what is being recorded, on a remote host and/or for another process.
Since JDK 11, there exists a FlightRecordingMXBean in OpenJDK that can control and download recordings remotely. This is how JDK Mission Control fetches recording data and configure events on a remote machine. In JDK 15, and earlier releases, a recording must be stopped before data can be read by a client.
In JDK 16, this restriction was lifted and JFR can now be used to monitor a remote host using a MBeanServerConnection.
4. Active, out of process (and over the network)
String host = "com.example";
int port = 7091;
String url = "service:jmx:rmi:///jndi/rmi://" + host + ":" + port + "/jmxrmi";
JMXServiceURL u = new JMXServiceURL(url);
JMXConnector c = JMXConnectorFactory.connect(u);
MBeanServerConnection connection = c.getMBeanServerConnection();
try (RemoteRecordingStream stream = new RemoteRecordingStream(connection)) {
stream.enabled("jdk.JavaMonitorEnter").withStackTrace();
stream.onEvent("jdk.JavaMonitorEnter", System.out::println),
stream.start();
}
The implementation of RemoteRecordingStream reads bytes of data from the FlightRecorderMXBean::readStream(long) method and writes it to disk locally, in chunks, similar to what the JVM does on the remote host. Another thread then parses the data on disk and dispatches events to the onEvent handlers. Once every second, new data becomes available to read.
To make sure the parser thread doesn’t read data before a data segment is complete, there is a size field in the chunk header that says how far into the file data can be read. Once new data arrive and a segment becomes complete, the field is updated. To make sure the size field is not modified, while being read, there is a protocol the parser must follow to avoid word tearing.
Once a chunk file has been read and its events dispatched, the file is removed from the client. To instead keep the data, two policies can be set, setMaxAge(Duration) and setMaxSize(long), to determine how long data should be retained.
The RemoteRecordingStream class is not just for streaming events, but also for migrating the disk repository to another host. If the monitored application crashes, and the disk repository files on the host are removed, they are still available on the machine where RemoteRecordingStream operates. Plan is to add a dump method to RemoteRecordingStream, so a file can easily be extracted if something goes wrong.
Streaming event metadata
A complication with streaming events from another process is the lack of access to event metadata. When streaming in process, the FlightRecorder::getEventTypes() method can be invoked to get a list of all registered event types.
Without knowledge of the event types, it’s not possible to determine the field layout, or which events that can be enabled/configured.
To remedy the situation, a new method was added to the EventStream interface:
void onMetadata(MetadataEvent);
The MetadataEvent carries a list of all registered event types and the two configurations, “default” and “profile”, that comes with the JDK. The MetadataEvent is sent before any onEvent handler is invoked. If a new event type is registered/unregistered, an updated MetadataEvent is sent.
To see the RemoteRecordingStream class in action, there exist a small single-file program called Health Report.java that subscribes to events and prints them to standard out.
Usage:
$ java HealthReport.java com.example:7091