2.1.1 UNIX signals
2.1.2 Windows signals

3.1.1 Time and date of the javacore
3.1.2 Type of signal and location of the JVM process at the time of the javacore
3.1.3 Java fullversion
3.1.4 Current thread details

3.2.1 Signal handlers
3.2.2 Loaded libraries
3.2.3 Java command line

back to top

2.1.1 UNIX Signals
The javacore displays the operating system signal that caused the javacore to be written. Some signals also produce a core file in UNIX operating systems. The JVM signal handler library is libhpi.a.

Following are some of the most commonly seen signals in javacore files:

• SIGQUIT
  This signal is sent when a kill -3 command is issued against the JVM process. This signal typically does not end the JVM process. It generates a thread dump (javacore) for diagnosing a potentially hung JVM process.
  
• SIGILL
  This is the equivalent of a kill -4 command. This means an illegal instruction was executed. This can mean a corruption of the code segment or an branch that is not valid within the native code. This signal often indicates a problem caused by JIT-compiled code.
  
• SIGSEGV
  This signal is equivalent to a kill -11 command. It indicates an operation that is not valid in a program, such as accessing an illegal memory address. This is typically indicative of a programming problem in one of the native libraries.

For more information on signals in WebSphere Application Server, refer to the technote What is a signal and why does this matter for WebSphere Application Server.

2.1.2 Windows Signals
The signals that Windows uses are entirely different than those used by UNIX. Here is a list of some of the operating system signals produced on Windows. Some signals also produce a user.dmp file. The JVM signal handler library is hpi.dll.

Signal Name	Description	JVM action
Memory access error	Invalid memory address	javacore and abort
Illegal access error		javacore and abort

3.1.1 Time and Date of the javacore
Every javacore file has a line that indicates the time and date of when the javacore file was produced. In the older javacore format, the first line of a javacore file identifies when the javacore was taken. In the newer javacore format, there is a line that has the tag, 1TIDATETIME and the tag Date: that indicates the date and time when the javacore was written.

Diagnostic Step
Verify when the javacore file was written to determine if it is relevant to problem being analyzed.

3.1.2 Type of Signal and Location of the JVM process at the time of the javacore
Every javacore has a line that indicates the signal that caused the javacore to be produced. In the older javacore format, this is the next line after the date in the javacore file. In the older javacore file format, the line that identifies the signal often identifies the library in which the signal was generated (if it is possible for the javacore processing to identify it). In the sample javacore output shown, the signal that generated the thread dump is a SIGSEGV, which means that the JVM process ended abnormally. The library in which the crash occurred is libdb2jdbc.so.

Sample header from the old javacore format

Tue Mar  9 18:18:31 2004 SIGSEGV received at 0xd44350a8 in /usr/lpp/db2_07/java12/libdb2jdbc.so. Processing terminated. J2RE 1.3.1 IBM AIX build ca131-20020722

In the newer javacore file format, the line with the tag 1TISIGINFO contains the signal that caused the javacore. It is important to understand what signal caused the javacore. This determines how to interpret the remaining information in the javacore file.

For the newer javacore format, the library that issued the operating system signal is identified in the line with the tag 1XHSIGRECV.

New format javacore

NULL           ---------------------------------------------- 0SECTION       TITLE subcomponent dump routine NULL           =============================== 1TISIGINFO     signal 11 received 1TIDATETIME    Date:                 2004/04/06 at 16:04:16 1TIFILENAME    Javacore filename:    /export/home/was4/wasapps/appvmc01/vmclaimt1/workdir/javacore499836.1081281856.txt NULL           ---------------------------------------------- 0SECTION       XHPI subcomponent dump routine NULL           ============================== 1XHTIME        Tue Apr  6 16:04:16 2004 1XHSIGRECV     SIGSEGV received at 0xd2782a88 in /opt/WebSphere4/AppServer/java/jre/bin/libjitc.a. Processing terminated. 1XHFULLVERSION J2RE 1.3.1 IBM AIX build ca131-20030630a

Diagnostic Step: Examining the Signal type and location
First, note the signal type. Does the signal indicate a JVM crash? Does the signal information state it was due to an operator action? If the javacore is written to diagnose a hung JVM process, the steps you take are very different than if the javacore is due to a JVM crash.

Next, if it is a signal that indicates a JVM crash, examine this line to view the signal that caused the javacore to be written and to see if it identifies the library that caused the JVM crash. Contact the company that developed the library to pursue a fix. If the library is a JVM native library, a Java SDK upgrade might resolve the problem. If the signal information does not identify the library that caused the crash, you must examine the Current Thread Details to see if you can use determine the library that caused the crash.

For example, the following message indicates a crash in a JVM library and should be pursued with WebSphere Application Server Support.

Crash in the libjitc.a library

1XHSIGRECV  SIGSEGV received at 0xd2782a88 in /opt/WebSphere4/AppServer/java/jre/bin/libjitc.a. Processing terminated.

This crash shows a JVM crash in a DB2® library and should be pursued with DB2 support.

Crash in a DB2 runtime library

SIGSEGV received at 0xd44350a8 in /usr/lpp/db2_07_01/java12/libdb2jdbc.so. Processing terminated.

back to top

3.1.3 Java Fullversion
The javacore identifies the Java fullversion of the JVM that produced the javacore. In the older javacore format, this line is the one after the one with the operating system signal. The line in the example above that reads J2RE 1.3.1 IBM AIX build ca131-20020722. In the new javacore format, the Java fullversion is in the line with the tag 1XHFULLVERSION.

back to top

3.1.4 Current Thread Details
The current thread is the thread that is running when the signal is raised that causes the javacore to be written. The Current Thread Details shows the Java and native stacks of the current thread. Since these are stacks, the most recent calls are at the top of the stack.

• The Java stack shows the Java method calls that are made by the current thread.
• If there is a native stack, this contains the most recent events that the thread executed. As the name implies, these are events in native code (libraries) called by Java.

Diagnostic Step
If the library is not identified by the signal information, you must examine the current thread details to see if the native stack indicates in which library the current thread was processing at the time of the JVM crash. If the signal is one that is used for an abnormal process termination, that is, SIGSEGV or SIGILL, the current thread details show the sequence of calls that caused the signal to be generated. For a JVM crash, if the line that identifies the signal is not able to identify the failing library, the native stack trace in the current thread details might be useful to identify the library.

In the two examples here, the location of the crash was already determined by the javacore processing, but this is not always the case.

Example 1

This is the current thread for the SIGSEGV in libdb2jdbc.so shown above. The current thread shows none of the Java stack; it might be that the javacore processing could not retrieve the information for the Java stack. The native stack shows that a jni_NewString call was made that caused the SIGSEGV to occur; the call to jni_NewString was made from Java_COM_ibm_db2_jdbc_app_SQLExceptionGenerator_SQLStatementError, which is a DB2 Java method.

Current Thread Details ----------------------  "Servlet.Engine.Transports:253" sys_thread_t:0x7D4C36C8 ----- Native Stack ----- at 0xD3FF7A34 in jni_NewString at 0xD44350A8 in Java_COM_ibm_db2_jdbc_app_SQLExceptionGenerator_SQLStatementError

Example 2

In this example, the current thread is in the libjitc.a library, which is generating native code for a Java method. This is not obvious from the first three lines of the native stack, but the fourth call down shows that it is the JIT compiler is involved.

1XHCURRENTTHD  Current Thread Details NULL           ---------------------- 2XHCURRSYSTHD  "EntigoAppsStarter" sys_thread_t:0x59AF8650 3XHNATIVESTACK Native Stack NULL           ------------ 3XHSTACKLINE   at 0xD2782A88 in dataflow_arraycheck 3XHSTACKLINE   at 0xD27226A0 in bytecode_optimization_driver 3XHSTACKLINE   at 0xD27251CC in bytecode_optimization 3XHSTACKLINE   at 0xD2685140 in JITGenNativeCode 3XHSTACKLINE   at 0xD26AB774 in jit_compile_a_method_locked 3XHSTACKLINE   at 0xD26ACD24 in jit_compiler_entry 3XHSTACKLINE   at 0xD26AD284 in _jit_fast_compile

In this case, a workaround is to skip JIT compiling of this method. A likely resolution is to upgrade the Java SDK to upgrade the libjitc.a library.

Note: For a JVM crash, a formatted core file or user.dmp file is also very helpful for diagnosing the cause of the problem.

3.2.1 Signal Handlers
This section shows you the Signal Handlers and the Environment Variables that are in use by the JVM process. A signal handler is a library that is defined to handle a signal if and when and it occurs. These can be programmatically defined; a library can register itself to handle a signal, intercept it and process it, and prevent it from being handled by the JVM process.

Signal Handler Info in new javacore format

1XHSIGHANDLERS Signal Handlers NULL           --------------- 2XHSIGHANDLER SIGHUP  : ignored 2XHSIGHANDLER SIGINT  : intrDispatchMD (libhpi.a) 2XHSIGHANDLER SIGQUIT : intrDispatchMD (libhpi.a) 2XHSIGHANDLER SIGILL  : intrDispatchMD (libhpi.a) 2XHSIGHANDLER SIGTRAP : JITSigTrapHandler (libjitc.a) 2XHSIGHANDLER SIGABRT : intrDispatchMD (libhpi.a) 2XHSIGHANDLER SIGEMT  : intrDispatchMD (libhpi.a) 2XHSIGHANDLER SIGFPE  : intrDispatchMD (libhpi.a) 2XHSIGHANDLER SIGBUS  : intrDispatchMD (libhpi.a) 2XHSIGHANDLER SIGSEGV : intrDispatchMD (libhpi.a) 2XHSIGHANDLER SIGSYS  : intrDispatchMD (libhpi.a) 2XHSIGHANDLER SIGPIPE : ignored 2XHSIGHANDLER SIGUSR2 : JITSigUSR2Handler (libjitc.a) 2XHSIGHANDLER SIGTERM : intrDispatchMD (libhpi.a)

Diagnostic Step
You can use the signal handler information to determine if a signal handler is intercepting a signal and preventing a javacore from being produced.

Note: libhpi.a is the normal JVM library for handling signals on AIX and Linux; it is the library that produces the javacore.

If a signal handler is preventing the javacore from being produced, then the wscp or wsadmin utility might be used to dump the threads and get a javacore. Furthermore, signal handlers can be disabled for the sake of diagnosing a problem. If you know that a product uses a signal handler, as IBM MQ® does, you can find out from product documentation how to disable the signal handler.

3.2.2 Loaded Libraries
This section of the javacore shows the native libraries that are loaded by the JVM.

Loaded library output for the libjitc.a library

2XHLIBNAME /opt/WebSphere4/AppServer/java/jre/bin/libjitc.a 3XHLIBSIZE         filesize    : 2731317 3XHLIBSTART        text start  : 0xD264A000 3XHLIBLDSIZE       text size   : 0x233D6C 3XHLIBLDORG        data start  : 0x524AE498 3XHLIBLDDATASZ     data size   : 0xF71C

Diagnostic Step
This can be useful in determining which version of a library is loaded in the event that there is more than one copy of the library installed on a system.

It can also be used to determine the location of a crash if the library name is not identified. This requires looking at the Text start and text size lines. In the new format javacore attached to this document, the offset of the crash is 0xd2782a88. The library whose text range contains the address offset is the library in which the signal occurred that produced the javacore. For the libjitc.a, its text range is 0xD264A000 to 0xD287DD6C.

3.2.3 Java Command Line
The javacore contains the command line that started Java. In the older javacore format, this is in a section entitled, System Properties. In the newer javacore format, the command line that started Java is identified with the tag 1CICMDLINE. This is in the CI subcomponent dump routine.

Use the Java command line to determine if the javacore is for a WebSphere Application Server Java process, and if so, which WebSphere Java process it is for: an Application Server process, an Administrative Server process, a jmsserver, a node agent, or some other WebSphere Application Server process.

back to top