Few days ago, I created few notes about Confluence and Jira. I follow Atlassian Docs and few things from my own experience. This post is only summary of all of that what I found at internet and what I tried to do as best practices. I hope you will find here some new suggestions and maybe it will help you to solve your problems. Everybody knows that these applications are really robust and in large corporates they are under huge pressure.
Ok, let’s start!
JVM & OS
JVM (Java Virtual Machine) is heart of application. If you starting Atlassian applications, they are written in Java and runs in JVM. Of course you know about setenv.sh or setenv.bat files in conf directory. There is possible to define parameters for runnning JVM. Three most common parameters are:
- Xms – the minimum size of the heap
- Xmx – the maximum size of the heap
- XX:MaxPermSize – the maximum size of PermGen
I don’t want to lost time to write their definitions, because you are able to find a lot of them on internet. But there is a good point, what I always did for bigger instances of Confluence or Jira.
ADVICE: Set Xms and Xmx to same value.
Reason: If you know that your usage of heap is around 70%-85%, it is OK. You have appropriately heap size. You are able to set minimum size of the heap as the same value as maximum (because you know that it will need it anyway) so why lost time and resources for allocating bigger size during running? These applications are used daily so they will need this size of heap always.
ADVICE: Raise your heap size by 512m blocks and PermGen size by 128m blocks.
ADVICE: Never set more heap size and PermGen by bigger values!
Reason: If you think that you add a lot of memory to this spaces and you will be OK for a long time, that is not true! If you let a huge space for them, there is invalid cleaning of objects in them and final performance will be lower! Take a care to them and watching their states.
ADVICE: Keep eough a RAM size! With low RAM, server start swapping heap data to harddrive-> generated more IOps -> slower application.
ADVICE: Proxying Atlassian server applications with Apache HTTP Server. How to do it and others information are on their site.
Performance problems are very often caused by wrong database setting or by database connection pool. Check these settings, these information are from official Atlassian docs.
ADVICE: Check collation (MSSQL) and character encoding. It should be UTF-8.
- DON’T use Tomcat 6 and older (DBCP (database pool) libraries in Tomcat were updated causing ClassCastExceptions (LOB))
- JDBC drivers ->Use only thin version of driver.
- Cas Sensitive Collation required!
If you want to set your own query timeout, follow these instructions:
Extract databaseSubsystemContext.xml from the confluence-x.x.x.jar that is in confluence/WEB-INF/lib/, and put a copy in confluence/WEB-INF/classes/. Edit confluence/WEB-INF/classes/databaseSubsystemContext.xml to add the defaultTimeout property to the “transactionManager” bean:
<bean id="tenantedTransactionManager" class="org.springframework.orm.hibernate.HibernateTransactionManager" plugin:available="true"> <property name="sessionFactory" ref="sessionFactory"/> <property name="defaultTimeout" value="120"/> </bean>
Database connection pool
Check your database connection pool. This pool defines cache for database connection for reuse.
ADVICE: Set database connection pool to value that matches 60-80% of the number of threads that will be used in peak time.
- Direct JDBC: confluence.cfg.xml ->
- Data source: conf/server.xml ->
<Resource name="jdbc/confluence" auth="Container" type="javax.sql.DataSource" maxActive="50"/>Note: For tomcat8 it is maxTotal parameter (in earlier versions it is maxActive)
Confluence Tips (From official Atlassian Site)
Check the ‘effectiveness‘ versus the ‘percent used’. A cache with a low percent used need not have its size lowered; it does not use more memory until the cache is filled.
Cache configurations are stored in
For Confluence Data Center (clustered) it can be found in
<confluence-shared-home>/config/cache-settings-overrides.properties (in the shared home directory for the cluster).
Content Objects cache (com.atlassian.confluence.core.ContentEntityObject) should be set to at least 20-30% of the number of content entity objects (pages, comments, emails, news items) in your system. To find the number of content entity objects, use the query
select count(*) from CONTENT where prevver is null.
Content Body Mappings cache (com.atlassian.confluence.core.ContentEntityObject.bodyContents) should be set to at least 20% of the number of content entity objects (pages, comments, emails, news items) in your system. To find the number of content entity objects, use the query
select count(*) from CONTENT where prevver is null.
Embedded Crowd Internal User cache (com.atlassian.crowd.model.user.InternalUser) should be set to the number of users you have in the internal directory. You can discover this number by using the following SQL:
SELECT COUNT(*) FROM cwd_user u JOIN cwd_directory d ON u.directory_id = d.id AND d.directory_name = 'Confluence Internal Directory'.
Embedded Crowd Users cachecom.atlassian.confluence.user.crowd.CachedCrowdUserDao.USER_CACHE should be set to the number of rows in the cwd_user table.
SELECT COUNT(*) FROM cwd_user u;
Space permissions by ID cache (com.atlassian.confluence.security.SpacePermission) should be set to the number of space permissions in your deployment (a good rule of thumb is 20 times the number of spaces). You can find the number of space permissions using the query
select count(*) from SPACEPERMISSIONS.
Index optimisation is really important. If you receive error like Lock Timeout Waited 30000 miliseconds, it is good point to start reindexing.
Permission checks are expensive and if you haven’t allow anonymous access, then replace jira-users with the ‘Anyone’ permission.
Scheduled Jira reindexing
If you want to set scheduled Jira reindexing, you are able to create a script where you will call REST API
If you will call it without type parameter, its perform BACKGROUND indexing (it is default action).
Then you are able to set script as executable by
chmod +x command and set CRONtab to execute new job by command:
crontab -e and add new line with CRON expression, for example:
0 1 * * sun. This expression set running at 01:00 AM every Sunday (values order is: minute – hour – day of month – month – day of week). More about CRON expression here!
Script for running scheduled Jira Reindex will be posted later in next post for Jira 😉