BIRT Exchange Forum: Virgil - Viewing Profile

Soon after the MongoDB noSQL-style database system started becoming popular, the BIRT community started asking for ways to get at that data. The good news for developers using BIRT, is that BIRT was designed with data source extensibility in mind. There are several ways to connect to MongoDB as a data source within BIRT. This post will show how to connect to MongoDB with your existing BIRT version, plus several screenshots from the new MongoDB ODA driver being added to the next major release of BIRT due out this summer.

BIRT Scripted Data Source Connection to MongoDB
The BIRT Scripted Data Source is like the ‘silver-bullet’ into any data source. If you cannot connect to your data with one of the built-in BIRT data sources, then the Scripted Data Source may be the answer. It uses the server-side RHINO Javascript engine allowing you to write a bit of Java or Javascript as a simple wrapper to your data. As long as you can get to your data in your application, then it’s likely the Scripted Data Source can get that data into BIRT.

To create a connection to MongoDB, you first need to add the MongoDB JAR to your project classpath. Now, you just add a bit of Javascript to the following methods to make it work.

Create connection and query in the open method

importPackage(Packages.java.lang);
importPackage(Packages.com.mongodb);
importPackage(Packages.java.util);
importPackage(Packages.org.bson);

m = new Mongo( "localhost" , 27017 );
db = m.getDB( "scratch" );
col = db.getCollection("zips");

query = new BasicDBObject();
query.put("state", "AL");
cursor = col.find(query);

The fetch method will get called repeatedly while you return true

if( cursor.hasNext() ){
var jsonStr = new String(cursor.next());
row["city"] = jsonStr.split(",")[1].split(":")[1].replace("\"", "");
row["zip"] = jsonStr.split(",")[2].split(":")[1].replace("\"", "");
row["state"] = jsonStr.split(",")[6].split(":")[1].replace("\"", "").replace("}","");
return true;
}else{
return false;
}

The close method is used to discard objects

cursor.close();

Community Created BIRT Plug-Ins for MongoDB
Part of BIRT’s extensibility allows for external plug-ins to be dropped into existing versions of BIRT. There have been several community developed BIRT plug-ins to access MongoDB data created over the years. At the time of this post, a quick search in the BIRT Exchange DevShare shows 3 different MongoDB ODA Driver plug-ins to choose from.



New MongoDB ODA Driver for BIRT
The next major release of BIRT will include a new data driver for accessing MongoDB. This means, instead of creating your own connection to MongoDB in a scripted data set, you can now simply select MongoDB from a list of available choices.



Next, fill out the database credentials and connection information. This information can be hard-coded in each specific field or the connection information can be dynamically generated at runtime using BIRT’s Property Binding feature.



Once you have created a connection, you can now create a new Data Set to query for specific data. BIRT supports the MongoDB operations that return data. This includes the Query commands, the Aggregate command, MapReduce, and select Database Commands that return data.

Creating a MongoDB Data Set in BIRT


The new MongoDB Data Set dialog allows you to choose the data Collection and fields you want returned from the query. Java Objects can be flatted out to bring in just the data elements, or the entire object can be brought back for further processing. Select Finish and we can now preview the results of the MongoDB query.



Using a MongoDB MapReduce Command in BIRT



Using a MongoDB Aggregate Command in BIRT


Using a MongoDB Database Command in BIRT


BIRT supports a subset of the MongoDB Database Commands that return data. The following is a list of the supported database commands from within BIRT.

buildInfo, collStats, connPoolStats, count, cursorInfo, dataSize, dbStats, distinct,
eval with nolock: true geoNear, geoSearch, getLastError, getLog, getPrevError, group, isMaster, isdbgrid, listCommands, listDatabases, listShards, ping, printShardingStatus, replSetGetStatus, serverStatus

Summary
There are several ways to connect to MongoDB no matter what version you are using. The new MongoDB ODA driver will make it’s first appearance in BIRT version 4.3 Milestone 7 due out soon.

According to Wikipedia, Big data is a collection of data sets so large and complex that they are difficult to process using on-hand database management tools or traditional data processing applications. Big data usually includes data sets with sizes beyond the ability of commonly used software tools.

It’s a good thing BIRT is not your traditional data processing application!

Eclipse BIRT was built with data source extensibility in mind. BIRT does this by leveraging the Eclipse Data Tools Project (DTP) and more specifically, the Open Data Access (ODA) framework. This framework allows new data sources, like recent big data sources, to be easily added to BIRT as needed. This post walks through creating a connection to Hadoop in order to visualize the data within BIRT.

Using HQL to query Hadoop data
BIRT provides an out-of-the-box driver that allows access to Hadoop Data through Hive using Hive Query Language (HQL). Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. To retrieve data from Hadoop, you write a query in Hive
Query Language (HQL). HQL supports many of the same keywords as SQL, for example
SELECT, WHERE, GROUP BY, ORDER BY, JOIN, and UNION.

A Hive query is executed by a series of automatically generated MapReduce jobs.
Alternatively, you can use the TRANSFORM statement to specify scripts that translate into
MapReduce functions in Hadoop. These scripts can be written in virtually any programming language. For example, the following HQL query specifies the script file mytest.py, written in the Python programming language.

SELECT
TRANSFORM (userid, movieid, rating, unixtime)
USING 'python mytest.py'
AS (userid, movieid, rating, weekday)
FROM u_data

Creating an HQL query in BIRT
To create a new query, select Hive Datasource from the New Data Source wizard
and enter the connection properties, as shown in the figure below.




Next, choose Manage Drivers and add the Hive client JAR files. You
only need to do this once.




Now, you can create a data set by writing an HQL query. If your query uses TRANSFORM
statements that reference script files, use the Add File Statement property to add files to the
Hadoop distributed cache. Type a semicolon-separated list of Add File commands. This property can be overridden by the data source or data set, using property binding or script. Type the HQL query in the query text area of the data set editor as shown below.


Complex HQL Subquery Example


Get JSON Object Example


Regular Expression Example


HQL Hints Example

Getting the Important Data to Stand Out
You can create multiple data sets using the same steps above, even joining data sets within BIRT. Once you have your big data connections and queries defined, you can start using the data to define your report within BIRT. At this point, you can simply drag your data sets onto the report canvas and start formatting.



But, being able to store more data brings its own new sets of challenges. The more data collected typically means more data that needs to be analyzed and displayed. This means, the important data really needs to stand out. BIRT supports this with several out-of-the-box features.

Highlighting
With the Highlighting feature in BIRT, you can set up formatting rules that are based on expressions. You can create simple to quite complex expressions in order to highlight the data. Highlighting can be added to grids, tables, columns, rows, data elements, labels, charts, and images.



Visibility
The Visibility feature allows you to use expressions to decide which areas of BIRT are visible. This is quite useful for allowing certain groups of people to see only their data but can also be used to hide whole areas based on the data discovered. Visibility can also be applied to grids, tables, columns, rows, data elements, labels, charts, and images, but you can also have seperate visibility rules based on the final output, like PDF, HTML, etc.



Try it Yourself
The Hive/Hadoop data source has been available since BIRT 3.7 and can be downloaded as part of the Eclipse BIRT Designer from eclipse.org/birt or BIRT Exchange.

The popular Getting Started with BIRT DZone Refcard has been updated. The BIRT DZone Refcard is the 50,000 view of the BIRT project from a developers perspective. It briefly explains the BIRT project in terms of components, select capabilities and features. It's just enough information to get you interested in BIRT and just enough detail to get you started.

The latest version has been updated to support BIRT 3.7 and includes additional information on BIRT Syles, Expressions and Scripting.

Download the BIRT DZone Refcard from http://www.birt-exch...dzone-refcardz/

BIRT Exchange will once again be sponsoring the EclipseCon 2012 Developer Conference. This year the conference will be held in Reston, VA (Washington DC area) from March 26th-29th.

There are 3 BIRT Sessions in this years EclipseCon conference program:

What is BIRT? Quick Jumpstart (presented by Virgil Dodson)
This session introduces BIRT and looks at how you can leverage BIRT to quickly create data-driven reports, web pages, and compelling information visualizations using its visual drag-and-drop design environment. This session provides a great orientation to the BIRT project and technology and how it might benefit your applications.

Extending BIRT with Plug-ins (presented by Michael Williams)
This session will show off several new plug-ins created by the community from a recent BIRT plug-in contest and also show step-by-step use of the ODA wizard to create a new BIRT data source and show where to find current plug-ins in the source code to use as a starting point for creating your own BIRT plug-ins faster. Attendees will leave this talk with an understanding of BIRT plug-ins and the various ways to extend BIRT.

Introduction to BIRT (by John Ward)
This tutorial will walk users through the BIRT environment and guide them through building their first few reports using the BIRT reporting tool

BIRT and Big Data
Pick up a whitepaper in the BIRT Exchange booth with step-by-step instructions on how to connect, access and display Hadoop data using HIVE Query Language (HQL) within BIRT-based applications.

Contests and Prizes
There will be lots of random giveaways to booth visitors, and we’ll announce a developer contest where we’ll be awarding three (3) new iPads.

After the success of our recent BIRT plug-in contest, we started thinking about another contest to keep the great BIRT community contributions coming in. We wanted a contest with a low barrier of entry allowing more users of BIRT, even new ones, to get involved. What we came up with is the 'BIRT Style Contest'. The contest allows BIRT users to show off their creative side by submitting a BIRT style example to the BIRT Exchange DevShare or Marketplace.

Actuate will award a Kindle Fire to each of the best three style examples submitted to either DevShare or the BIRT Exchange Marketplace between January 9, 2012 and February 9, 2012.

Styles are used within BIRT designs allowing you to create richly formatted web content. These styles can be applied directly within the BIRT Designer, imported from existing CSS stylesheets, and referenced at runtime to match your existing web application look and feel. Styles can also be applied dynamically based on parameters or scripting creating a customized experience for the user.

Some BIRT style examples might include:

• Clever use or colors (seasons, team colors)
  
• Style applied dynamically based on time of day, or year.
  
• Style applied based on external data like weather or regional info
  
• Your unique BIRT style example

Any DevShare submission that makes use of a BIRT style will be considered for this contest.

Contest rules, details and more suggestions can be found on the Contest Details and Examples page.

I look forward to seeing some more great contributions!