Glassfish may be the best J2EE Application Server out there. It is stable, it is fully j2ee 1.5 compatible, it runs all technologies and it is open source! I love this application server.
Here are some tips that will make it respond faster and increase its capacity in requests per second.
1. Disable application auto-deployment and dynamic class reloading :
Stand-Alone Instances > server (Admin Server) > Advanced tab
-> Auto Deploy --> uncheck
-> Reload --> uncheck
2 Disable dynamic JSP reloading :
Edit "default-web.xml" inside the config directory of each instance and change the init-param development to false for the org.apache.jasper.servlet.JspServlet (instance restart is required):
3. Minimize logging.
Logger writes to disk and that is very expensive. So you can rearrange logging levels to "SEVERE" for all the loggers (since this setting is dynamic and you can change it to diagnose without restarts, Cool!).
Admin Console > Logger Settings page > Log Levels tab
Further log minimizing:
Tail your server.log at peak hours and watch carefully at the messages. Catch the loggers that print unnecessary messages and either speak to the programmer to minimize the messages of their applications or you can immediately adjust the logging level for that logger subsystem!
Add a new property with the logger name and set the level to SEVERE or even OFF!!!
4. Tuning HTTP File Caching in memory for faster response to static resources:
Configurations > config-name > HTTP Service (HTTP File Cache)
Globbally : true
Max Files Count : 128 - 512 (is a good start for small applications but it really depends on the applications static resources)
Max Age : 86400 (1 day) - 604800 (7 days) is a good start for non dynamic reloading applications (see tip No 3)
5. JVM Parameters:
Configurations > config-name > JVM Settings (JVM Options).
-server [maximum program execution speed by advanced optimized compilation]
-XX:+UseConcMarkSweepGC [using the Concurrent Mark Sweep garbage collector can cause a drop in throughput for heavily utilized systems, because it is running more or less constantly, but it prevents long pauses, so it is best for real time applications]
-XX:+DisableExplicitGC [Disable explicit full gc collections (System.gc() calls) since it would only interfere with the garbage collection algorithms and create big pause times]
-Xms=-Xmx [having the same starting and maximum heap memory will avoid spending time on any kind of unnecessary resizing of the heap memory]
-Xmn [Set at most half of the heap memory, since the garbage collection here should faster, more often and contains short lived objects. A good start is 1/5 of the heap size.]
-Xss128k [128k Stack Size is a very good start. If you get Stack Overflow error increase it by 128k at a time until you reach a point where you no longer get the error. You might even lower it to 64k (or lower) if your application is really lightweight, and then you will be able to serve more concurrent clients ]
-XX:SurvivorRatio=8 [survivor space and eden ratio will be 1:8. If survivor spaces are too small, copying collection overflows directly into the old generation. If survivor spaces are too large, they will be empty.]
-XX:MaxPermSize=-XX:PermSize [If you get an "java.lang.OutOfMemoryError: PermGen space", you need to increase this value, since the default is 64MB. If you set the initial size and maximum size to equal values you may be able to avoid some full garbage collections that may occur if/when the permanent generation needs to be resized.]
-XX:+CMSClassUnloadingEnabled [Enables the CMS Garbage collector (if you use it) to cleanup the PermGen space too. ]
-XX:+UseParNewGC [this parallel young generation collector can be used with the concurrent low pause collector that collects the tenured generation.]
-XX:ParallelGCThreads [If number of cpus is less than 8 then put the number of cpus else add (3 + (5/8) * (number of cpus)) ]
-XX:TargetSurvivorRatio=90 [Allows 90% of the survivor spaces to be occupied instead of the default 50%, allowing better utilization of the survivor space memory. ]
-XX:MaxTenuringThreshold=30
-Djava.awt.headless=true
-Dcom.sun.enterprise.server.ss.ASQuickStartup=false
JVM Parameters example:
-server -Xmx2g -Xms2g -Xmn800m -Xss128k -XX:SurvivorRatio=8 -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -XX:+UseParNewGC -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=30 -Djava.awt.headless=true -Dcom.sun.enterprise.server.ss.ASQuickStartup=false
6. Tuning Linux :
Start by checking system limits for file descriptors with this command:
$ cat /proc/sys/fs/file-max
8192
The current limit shown is 8192. To increase it to 65535, use the following command (as root):
$ echo "65535" > /proc/sys/fs/file-max
To make this value to survive a system reboot, add it to
/etc/sysctl.conf and specify the maximum number of open files permitted:
fs.file-max = 65535
Note: The parameter is not
proc.sys.fs.file-max, as one might expect.
To list the available parameters that can be modified using
sysctl:
$ sysctl -a
To load new values from the
sysctl.conf file:
$ sysctl -p /etc/sysctl.conf
To check and modify limits per shell, use the following command:
$ limit
The output will look something like this:
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 8192 kbytes
coredumpsize 0 kbytes
memoryuse unlimited
descriptors 1024
memorylocked unlimited
maxproc 8146
openfiles 1024
The
openfiles and
descriptors show a limit of 1024. To increase the limit to 65535 for all users, edit
/etc/security/limits.conf as root, and modify or add the
nofile setting (number of file) entries:
* soft nofile 65535
* hard nofile 65535
The character “
*” is a wildcard that identifies all users. You could also specify a user ID instead.
Then edit
/etc/pam.d/login and add the line:
session required /lib/security/pam_limits.so
On Red Hat, you also need to edit
/etc/pam.d/sshd and add the following line:
session required /lib/security/pam_limits.so
On many systems, this procedure will be sufficient. Log in as a regular user and try it before doing the remaining steps. The remaining steps might not be required, depending on how pluggable authentication modules (PAM) and secure shell (SSH) are configured.
Tune the TCP/IP settings :
Add the following entry to /etc/rc.local
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 60000 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 15000 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
Add the following to /etc/sysctl.conf
# Disables packet forwarding
net.ipv4.ip_forward = 0
# Enables source route verification
net.ipv4.conf.default.rp_filter = 1
# Disables the magic-sysrq key
kernel.sysrq = 0
net.ipv4.ip_local_port_range = 1204 65000
net.core.rmem_max = 262140
net.core.rmem_default = 262140
net.ipv4.tcp_rmem = 4096 131072 262140
net.ipv4.tcp_wmem = 4096 131072 262140
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_keepalive_time = 60000
net.ipv4.tcp_keepalive_intvl = 15000
net.ipv4.tcp_fin_timeout = 30
Add the following as the last entry in /etc/rc.local
sysctl -p /etc/sysctl.conf
Reboot the system.
Use this command to increase the size of the transmit buffer:
tcp_recv_hiwat ndd /dev/tcp 8129 32768
Make the OS to use swap file only on emergencies:
swappiness=0
swap off; swap on
7. Disable the Security Manager only if your application server is inside an intranet or inside a very well protected environment (If you are sure that no malicious code will be run on the server and you do not use authentication within your application, then you can disable the security manager). It is generally not recommended but it could provide a significant performance boost (since the security manager has expensive calls).
Configurations >
config-name > JVM Settings (JVM Options)
delete the option that contains the following text:
-Djava.security.manager
8. Performance monitoring tools:
jconsole : jmx instance monitoring
jvisualvm : will give you a clear view of the cpu utilization, the garbage collections, perm gen size and jvm options. It can also profile and create thread dumps.
Applications Manager : is a nice commercial monitoring tool. Has good support for GlassFish (There is a Free Licence for up to 5 Glassfish instances through jmx).
nmon : nice open source linux monitoring command line gui.
9. Disable monitoring if you have no problems to detect and resolve.
Configurations >
config-name > Monitoring
Change all levels to Low if you still need a little monitoring, or Off if your applications are running smoothly.
10. Update your software as often as your infrastructure availability/downtime permits. Update your jdk, since bug fixes and optimizations may increase the performance of your application server and make some weird errors go away! The same goes for the operating system, as well as the jdbc drivers or any other libraries your applications use.
11. Patch Glassfish 2.1.x with the latest Grizzly 1.0.x releases.
The Grizzly thread manager library is the heart of GlassFish. Since oracle is not planning to release another v2 release, this kind of patch is significant and relatively easy. Unless you decide to move on to v3 release!
First Check the current Grizzly version by setting the JVM property: "
-Dcom.sun.enterprise.web.connector.grizzly.displayConfiguration=true".
Make sure the web-container log level is INFO. Restart Glassfish instance and check server.log. You should see an output like this :
Grizzly 1.0.30 running on Mac OS X-10.5.8 under JDK version: 1.6.0_15-Apple Inc.
port: 8080
maxThreads: 5
ByteBuffer size: 4096
useDirectByteBuffer: 8192
maxKeepAliveRequests: 250
keepAliveTimeoutInSeconds: 30
Static File Cache enabled: false
Pipeline : com.sun.enterprise.web.portunif.PortUnificationPipeline
Round Robin Selector Algorithm enabled: false
Round Robin Selector pool size: 1
Asynchronous Request Processing enabled: true|#]
where Grizzly version is 1.0.30. Please note, that if you'll see a similar output, but without Grizzly version in it, it means that your version is older than 1.0.30, so a Grizzly upgrade is recommended. Download latest Grizzly 1.0.x binary file from:
http://download.java.net/maven/2/com/sun/grizzly/grizzly-framework-http/
and save it to a directory, for example :
/home/gfuser/grizzly/grizzly-framework-http-1.0.30.jar
Then set Glassfish prefix-classpath to "
/home/gfuser/grizzly/grizzly-framework-http-1.0.30.jar" to force Glassfish use the latest Grizzly classes instead of the embedded ones. Restart Glassfish and check the server.log again to confirm the success of the patch. Reset the web-container log level to SEVERE.
Further reading.
Java Tuning White Paper :
http://java.sun.com/performance/reference/whitepapers/tuning.html
Frequently Asked Questions about Garbage Collection :
http://www.oracle.com/technetwork/java/faq-140837.html
Tuning Garbage Collection :
http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html
Java HotSpot VM Options :
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
Sun Java System Application Server 9.1 Performance Tuning Guide :
http://download.oracle.com/docs/cd/E19159-01/819-3681/
Any comments, suggestions, problems or requests will be warmly welcomed.
p.s.
You can find me on
fiverr for more personalized requests on any java app server configuration problem or java error that you encounter, with deliverance of less than a day (true!) and money back guarantee if not satisfied.
