TalkNotes - The story of SonarQube told to a DevOps Engineer

This week I spoke at Bangalore DevOps meetup on the topic "The story of SonarQube told to a DevOps Engineer". I have started writing TalkNotes inspired from Martin Fowler.Unlike his detailed article my posts aims to help the audience better understand my slides. SonarQube is a open source code quality management platform. It was a 30 mins talk focused at the need, setup, CI Infrastructure and administration of the SonarQube to the DevOps community.

The story of SonarQube told to a DevOps Engineer from Manu Pk

I have started the talk with one of my favorite subject Technical Debt. We have also looked at some of the parameters which determines the quality are coding standards breach, duplication, lack of unit tests, bad distribution of complexity, Spaghetti Design etc... I have spoke about this in more detail at previous post. There are various existing tools that helps reduce the technical debt by improve the code quality. What was missing was a easier way of tracking these code rule violations. For examples I need to know how much debt was introduced or was cleaned up? As a developer how do you quantify improvement the which a particular code refactoring has brought to the team etc..

This is where Sonar come to your help. Sonar's rich feature set allows you to do these and more. Currently it can run the quality analysis on more than 20 languages including Java, C#, C/C++, PL/SQL, Javascript, PHP, Web, XML etc.... It stores the analysis results and the data is displayed through various dashboards. Further slides discusses the sonar platform overview and installation.

The below diagram shows the CI environment including SonarQube. The Hudson plugin for SonarQube can be configured by following the wiki.

Image Idea from this blog.

The best part of sonar is its documentation. This was the most comprehensive documentation I have read about any open source product. You just need their wiki page to get 99% of answers. Now that I have it configured hope to write more about it in the coming moths.

4 simple steps to migrate legacy projects from Ant to Maven

For some time we were thinking about migrating our build to maven from ant. It happened last month and was actually simpler than what we have anticipated. From my experience, here is a brief about the steps we have followed. Our application is a  is a enterprise web application build with multiple frameworks and technologies and is deployed as a single WAR.

1. Create maven project directory structure.

As told in the Maven user guide create the below directory structure. We have done it under a new folder for the project.

2. Move the files/folders keeping the SCM logs. 

Even though the folder structure is new the source files will be old ones! We want to keep the SCM logs while moving them to new locations. Remember to commit the folders created in step 1 before you start moving your files. If you use SVN, see this user guide or SO question on how to do this. Move the java source, unit/integration test and configuration resources to appropriate folders.

3. Create the POM and add dependencies

Most critical part in the migration is adding the dependencies in the POM. Start by adding the dependencies for the frameworks used in your application. Make sure you are adding the right version of the jars. You can find the version of the jar by reading the MANIFEST.MF file inside the META-INF folder of the jar. This will help if the versions are missing from the file name.

Any third party jars can be added to the maven repository as told here. If you are using very old versions of jar files some of them may not be available in maven repository.Here you can either try upgrading to newer versions or prepare local install as told before. Once you have added all the dependencies try building the application. Watch out for any major issues.

4. Make sure you haven't changed much in the WAR

Maven is a build tool. This means your WAR should not change. So, in the last step we compare both versions and make sure they are the same. Make sure you are on top of all the differences. Also, compare the jar files generated by maven and your existing files, Sync them by 

     - Adding <exclusions> to remove the unwanted jars

     - Add the dependencies for the missing jars

This can be a tiring tasks depending on the number of jars you have in your lib. But make sure that you have each one covered and know that why they exists in your app.

May  be this is a late post, most applications might have already been migrated by now. Anyways, better late than never! According to many experts Gradle is also a good choice as a build tool for your new project.

Why is tomcat a Webserver and not an Application Server

Many application developers do not focus much on the infrastructure on which their code runs. When it comes to web applications there are common confusions like what is the difference between webserver and applications server or when to go for a EAR vs WAR file deployment etc...

There are many good answers that differentiate between web servers and applications servers like this one. Most of the times the terms Web Server and Application server are used interchangeably. This article explains the working of a typical web server. Typically we get confused with the example of Tomcat Server (an example for a web server) having the capability to run the enterprise applications. So, tomcat is a web server or an application server? Let me tell you how I convinced my self regarding this.

Some time back I was struck with the question What's the difference between JPA and Hibernate on stack overflow. I did answer it, but one of the comment lead me to a more detailed understanding of the JavaEE spec and certified servers. If you can understand this then differentiating between the web server and application server is easy. During my investigations I got this article, which discusses the advantages of both.

A more detailed look in to the meaning JavaEE specification will throw some light in to our discussions. As we know specifications are set of rules. Simply put they contain the interface. Any JavaEE servers which needs to comply to spec needs to have the implementation of these interfaces. You can find the certified JavaEE servers list here. If you are deploying your enterprise applications (means you have JPA, EJB or some technology which is part of Java EE) to the a server which comply to JavaEE then the lib need not contain the API implementation jars. But these are needed if you are using a web server like tomcat for deployment.

For example, if you use JPA in your applications and deploying it to the  Jboss AS 7, then you need any additional jars in the lib. But the same application you want to deploy to the tomcat server then you need to have additional jars to lib that implements the JPA spec may be eclipselink or Hibernate. This is what makes JBoss AS 7 an application server and tomcat a web server. Another key difference is that we can not deploy an EAR file to tomcat, it could only handle WAR files.

Does the View in database reduce the query performance?

Not really!! Here is why...

Few days back, I was speaking about one of our newly created Filter API with server side pagination to my friend. In the implementation we were querying the view to get the data. Since the pagination was at the server side, at a time the query will return only 10 records. His question was since views internally going to use the tables for querying, will it end up being less performative? Will it create the result set of all data and then pick the top 10 items?

My initial reaction was to agree with question and confess we have an issue. But I was sure that this can not be the case. If this is so then we could have had severe issues. I am not much of a DB guru myself so went and did some research with others to figure out the story behind the scenes. By the way we use MS SQL Server 2008 and I am not sure if this descriptions applies to other database or not. 

The question to ask was. We know that the views are used to simplify other queries or standardize access to data. but do we compromise on speed while doing so? 

I got a related answer from SO. When an SQL statement references a nonindexed view, the parser and query optimizer analyze the source of both the SQL statement and the view and then resolve them into a single execution plan. There is not one plan for the SQL statement and a separate plan for the view. So it is fine to use the views for my above use case.

Further, I got to know about Indexed views from this discussions, which can further improve the performance. Microsoft has a nice documentation on Improving Performance with SQL Server 2005 Indexed Views.

A MindMap for Java Developer Interviews

Over the years I have been a panelist in many of the interviews for Java Developers. I have previously written a post titled Top 7 tips for succeeding in a technical interview for software engineers which covers few of the general guidelines. In this post I will share a mind map containing general topics covered in a Java developer interview. I have prepared this as a general reference for myself to remember the pointers and to keep a common standard across the multiple interviews.

XMind gives a nice listing of the map. You can find the map here. Here is Image which you can download and use.

Finally here is a old fashioned tabbed content list which is easier to copy paste.

Java-Topics
OOPs
Encapsulation
Abstraction
Inheritance
Interface - Abstract Class
Casting
IS-A vs HAS-A Relationships
Aggregation vs Composition
Plymorphism
Method overloading vs Method Overloading
Compile time vs Runtime
Threads
Creating threads
Multitasking
Synchronization
Thread Transitions
Marker Interface
Serialization
Clonnable
Shallow copy vs Deep Copy
Collections
Map, List and Set
Equals - Hashcode
Legacy - Synchronized Classes
JVM
Stack vs Heap Memory
Garbage Collection
JRE, JVM, JDK
Class loaders
Exception
Checked Vs Unchecked Exceptions
Exception handling best practices
try, catch, finally, throw, throws
APIs
Files
String - StringBuffer - String Builder
Java IO
XML
SAX Based & DOM Based
JAXB - Java API for XML Binding
Access specifier 
Access modifier 
public
protected
deafult
private
final
static
synchronized
abstract
transient
volatile
 Inner/Nested Classes
JavaEE Basics
Packaging the Applications
WAR
EAR
Basics
MVC
Servlets
Listeners
Lifecycle
JSPs
APIs
JPA
JAX-WS
SOAP, WSDL Webservices basics
Contract first vs
JAX-RS
RESTful  and its advantages

JSF


This is a work in progress and I hope to refine it further. Let me know if you have any comments.

On the way to a blogger - My Journey with this blog so far

TalkNotes - How can Agile help you in clearing the technical debt?

This post is a write up to my talk titled "How can Agile help you in clearing the technical debt?" presented at the Scrum Bangalore Meetup 2013. It was a short talk for 30 Minutes, So thought of a write up to elaborate the ideas on the slides.

How can agile help you in clearing the technical debt from Manu Pk

Defining Technical Debt

As per Wikipedia, "Technical debt metaphor referring to the eventual consequences of poor or evolving software architecture and software development within a code base". Technical debt is most understood when it is compared with Cost of Change of software. More Debt means the cost of introducing a change in your system is more.

The term Technical Debt was coined by Ward Cunningham in the year 1992. He used the term to explain to his manager about a particular refactoring which he wanted to do. Since he was working in the financial industry it was easy to communicate in terms of debt. Ward explains more about the metaphor here. The term is immensely popular in communication of the importance of code refactoring to the business and stake holders. Like financial debt technical debt is some times unavoidable but we should take steps to minimize it.

As you can see in the above image when you have technical debt in your system you could either pay the interest or pay back the principle. When we do regular stories we do bit of "extra" work than what ideally needed due to the technical debt. This is an example of the interest payment. It is important that we take some time and effort to pay back the principle by refactoring and cleaning up of the code.

Technical Debt in Agile Context

The below principle from the list of agile principles was the best one I could find which refers to clearing technical debt "Continuous attention to technical excellence and good design enhances agility". In agile projects working software has a lot of emphasis but this in no way implies that the intrinsic value of software can be compromised. As we push for more and more features it is important that we take time to look back at the health of the application. So, make make sure that your agile projects do not become Fragile as it grows.

There are lots of ways to deal with technical debt. I want to talk about 3 simple steps that we have tried in our projects.

1. Tools - Your must have defense

One of the major improvement with the agile development practices is the widespread usage of continuous integration (CI). Your CI environment should be configured to measure the health of the application. It should include the report from static code analysis and test coverage applications. SonarQube is a god platform where you can create a dashboard and see results from the tools mentioned above. It also has a plugin to view the technical debt. Apart from the tools you should be using good programming practices and guidelines.

2. Negotiating with Product Owner

The second techniques that we have tried to minimize technical debt is to negotiate with the product owner and take him on board in the technical improvements that we wanted to do.  You have to Use the opportunity of Change requests to clean up the functionality. We had a module in our application with lots of bugs. There was a major change request which was planned on that module. After much discussions we identified that it would be much easier to rebuild the module than fixing the issues. Once we rebuild the change request was 75% easier to implement compared to the old code.

This means that we have to evaluate the possibility of refactoring with every opportunity and go for it when the time is right. It is the professional responsibility of a developer to communicate dept of technical debt in the application and we should take all steps possible to minimize it. 

We can also have technical Stories in Release planning. Contrary to a user story, a senior developer should describe the success criteria of the story. This is another case where you need to get your product owner on board.

3. Trying out new Stuff

In our projects most things can be done multiple ways and we have to choose one of them. There are also situations where we are not sure about the complexity of the task performed and story point estimation is difficult. We can use spike to deal with such situations. A spike is an experiment that allows developers to learn just enough about something unknown in a user story, e.g. a new technology, to be able to estimate that user story. A spike must be time-boxed. This defines the maximum time that will be spent learning and fixes the estimate for the spike.

A daemon thread is a low priority thread that is running on the operating systems. We have used the same strategy to do a major change in our application. It was upgrading our frameworks (Spring, Hibernate and Tapestry) to their latest versions. Since it is a time consuming activity, we do not want to stop everything else and work on it. So, we choose to keep doing it as a low priority tasks and it took us 10 sprints to complete. Since the team continuously delivered features during this period it was very easy to get the buy in from product owner.

Useful Links:-

1. Martin Fowler's Talk

2. Martin Fowler on Technical Debt

3. Agile Architecture Emerges or Does It

4. Using Spikes

5. Steve McConnell’s Talk

6. http://www.ontechnicaldebt.com

7. Ward Explains Debt Metaphor

Note:- There are many ways to define and minimize technical debt. I have discussed few ways that we have successfully tried. You can go through the above links to find more information and choose the best methods that suits you.