Atlassian

Few tips for Atlassian Confluence/Jira administrators for better performance

Introduction

Few days ago, I created few notes about Confluence and Jira. I follow Atlassian Docs and few things from my own experience. This post is only summary of all of that what I found at internet and what I tried to do as best practices. I hope you will find here some new suggestions and maybe it will help you to solve your problems. Everybody knows that these applications are really robust and in large corporates they are under huge pressure.

Ok, let’s start!

JVM & OS

JVM (Java Virtual Machine) is heart of application. If you starting Atlassian applications, they are written in Java and runs in JVM. Of course you know about setenv.sh or setenv.bat files in conf directory. There is possible to define parameters for runnning JVM. Three most common parameters are:

  • Xms – the minimum size of the heap
  • Xmx – the maximum size of the heap
  • XX:MaxPermSize – the maximum size of PermGen

I don’t want to lost time to write their definitions, because you are able to find a lot of them on internet. But there is a good point, what I always did for bigger instances of Confluence or Jira.

 

ADVICE: Set Xms and Xmx to same value.

Reason: If you know that your usage of heap is around 70%-85%, it is OK. You have appropriately heap size.  You are able to set minimum size of the heap as the same value as maximum (because you know that it will need it anyway) so why lost time and resources for allocating bigger size during running? These applications are used daily so they will need this size of heap always.

ADVICE: Raise your heap size by 512m blocks and PermGen size by 128m blocks.

ADVICE: Never set more heap size and PermGen by bigger values!

Reason: If you think that you add a lot of memory to this spaces and you will be OK for a long time, that is not true! If you let a huge space for them, there is invalid cleaning of objects in them and final performance will be lower! Take a care to them and watching their states.

ADVICE: Keep eough a RAM size! With low RAM, server start swapping heap data to harddrive-> generated more IOps -> slower application.

ADVICE: Proxying Atlassian server applications with Apache HTTP Server. How to do it and others information are on their site.

Database

Performance problems are very often caused by wrong database setting or by database connection pool. Check these settings, these information are from official Atlassian docs.

ADVICE: Check collation (MSSQL) and character encoding. It should be UTF-8.

Oracle

  1. DON’T use Tomcat 6 and older (DBCP (database pool) libraries in Tomcat were updated causing ClassCastExceptions (LOB))
  2. JDBC drivers ->Use only thin version of driver.

MSSQL

  1. Cas Sensitive Collation required!

Query timeout

If you want to set your own query timeout, follow these instructions:

Extract databaseSubsystemContext.xml from the confluence-x.x.x.jar that is in confluence/WEB-INF/lib/, and put a copy in confluence/WEB-INF/classes/. Edit confluence/WEB-INF/classes/databaseSubsystemContext.xml to add the defaultTimeout property to the “transactionManager” bean:

<bean id="tenantedTransactionManager" class="org.springframework.orm.hibernate.HibernateTransactionManager" plugin:available="true">
  <property name="sessionFactory" ref="sessionFactory"/> 
  <property name="defaultTimeout" value="120"/>
</bean>

 

Database connection pool

Check your database connection pool. This pool defines cache for database connection for reuse.

ADVICE: Set database connection pool to value that matches 60-80% of the number of threads that will be used in peak time.

Setting locations:

  • Direct JDBC: confluence.cfg.xml -> <property name="hibernate.c3p0.max_size">30</property>
  • Data source: conf/server.xml -> <Resource name="jdbc/confluence" auth="Container" type="javax.sql.DataSource" maxActive="50"/>Note:  For tomcat8 it is maxTotal parameter (in earlier versions it is maxActive)

Confluence Tips (From official Atlassian Site)

Check the ‘effectiveness‘ versus the ‘percent used’. A cache with a low percent used need not have its size lowered; it does not use more memory until the cache is filled.

Cache configurations are stored in <confluence-home>/shared-home/config/cache-settings-overrides.properties.

For Confluence Data Center (clustered) it can be found in <confluence-shared-home>/config/cache-settings-overrides.properties (in the shared home directory for the cluster).

Content Objects cache (com.atlassian.confluence.core.ContentEntityObject) should be set to at least 20-30% of the number of content entity objects (pages, comments, emails, news items) in your system. To find the number of content entity objects, use the query select count(*) from CONTENT where prevver is null.

Content Body Mappings cache (com.atlassian.confluence.core.ContentEntityObject.bodyContents) should be set to at least 20% of the number of content entity objects (pages, comments, emails, news items) in your system. To find the number of content entity objects, use the query select count(*) from CONTENT where prevver is null.

Embedded Crowd Internal User cache (com.atlassian.crowd.model.user.InternalUser) should be set to the number of users you have in the internal directory. You can discover this number by using the following SQL: SELECT COUNT(*) FROM cwd_user u JOIN cwd_directory d ON u.directory_id = d.id AND d.directory_name = 'Confluence Internal Directory'.

Embedded Crowd Users cachecom.atlassian.confluence.user.crowd.CachedCrowdUserDao.USER_CACHE should be set to the number of rows in the cwd_user table. SELECT COUNT(*) FROM cwd_user u;

Space permissions by ID cache (com.atlassian.confluence.security.SpacePermission) should be set to the number of space permissions in your deployment (a good rule of thumb is 20 times the number of spaces). You can find the number of space permissions using the query select count(*) from SPACEPERMISSIONS.

Jira Tips

Index optimisation is really important. If you receive error like Lock Timeout Waited 30000 miliseconds, it is good point to start reindexing.

Permission checks are expensive and if you haven’t allow anonymous access, then replace jira-users with the ‘Anyone’ permission.

Scheduled Jira reindexing

If you want to set scheduled Jira reindexing, you are able to create a script where you will call REST API "${server}/rest/api/2/reindex?type=FOREGROUND".

If you will call it without type parameter, its perform BACKGROUND indexing (it is default action).

Then you are able to set script as executable by chmod +x command and set CRONtab to execute new job by command: crontab -e and add new line with CRON expression, for example:  0 1 * * sun. This expression set running at 01:00 AM every Sunday (values order is: minute – hour  – day of month – month – day of week). More about CRON expression here!

Script for running scheduled Jira Reindex will be posted later in next post for Jira 😉

Aleš Laňar
Senior Engineer Software ve společnosti CA Technologies

5 thoughts on “Few tips for Atlassian Confluence/Jira administrators for better performance

  1. Woա, ɑwеsome bloց lаyout! How long have you been bⅼοgging for?
    you made blogging look easy. The overall loⲟk ߋf your web site iѕ
    wonderful, let alone the content!

  2. “If you let a huge space for them, there is invalid cleaning of objects in them and final performance will be lower!” Please, can you back this claim up with any kind of evidence? It looks completely fabricated

    By the way, JDBC drivers are not “think” but “thin”

    1. Typo is fixed. Thank you!
      And now something more about JVM size and performance. This is not fabricated. Easiest is described it by simple example. GC have to keep object, cleaning them, etc. In real world, you can imagine that you are living in big house, but you still need only 4 rooms, but you have to still clean up others X rooms.
      A now little bit more professionally.
      If you have a big Java application, all new objects are created in Eden. And now imagine that you GC want to look throught it for space that cen be raclaimed. There are a lot of kinds of evidence which you can find on internet. But another one good point is that keep on mind that you dont want to allocate all machine RAM for JVM, others proccess can starts swaping and it is always performance kill.
      I don’t want to write again what is already written, so here are few links with good posts in my opinion:
      http://www.javaworld.com/article/2078731/java-platform/jvm-performance-optimization–part-5–is-java-scalability-an-oxymoron-.html
      https://docs.oracle.com/cd/E13222_01/wls/docs81/perform/JVMTuning.html
      http://crunchify.com/jvm-tuning-heapsize-stacksize-garbage-collection-fundamental/
      But if you want to discuss it here, I am open for this discussion 🙂

  3. This is really interesting, You’re a very skkilled blogger.
    I’ve joined your feed and lopok forward to seeking more of your
    antastic post. Also, I have shared your site in my social networks!

Leave a Reply

Your email address will not be published. Required fields are marked *