esProc Assists Report Development – Transpose Operation for Crosstab Creation

It’s difficult to handle unconventional statistical tasks using simply the reporting tool, like Jasper or BIRT, or SQL. One of the cases is that the source data don’t meet the crosstab’s requirements and thus need to be transposed for display. Having powerful computing engine to process structured data and being integration-friendly, esProc is very useful in assisting the handling of the case. An example will be cited to explain the transposition for designing a crosstab report.

The database table booking holds the summary data of goods orders in every year with four fields that include the year and three types of order status. Some of the data are as follows:

The report table should display the order information of the specified year and the previous one, in which the row headers are the three types of order status and the column headers include years and the growth rate for each order status in the specified year. The measurement is the order data of the current year. The layout and appearance of the report is as follows:

The difficulty of creating this crosstab report is that the source data cannot be used directly and the values in the summary column need to be computed dynamically based on relative positions. However the difficulty will be significantly reduced if the column and row data in the source table can be rotated and summary values are computed, as shown below:  

Then use esProc code to compute the necessary data for the report:

A1=yearBegin=yearEnd-1

yearEnd is a user-defined report parameter representing the specified year, such as the year of 2014. A1’s code is used to determine the previous year, which can be defined as yearBegin for the convenience of reference.

This line of code retrieves data of the specified year and the previous one from the database. myDB1, the data source name, points to MySQL. query function can not only execute the SQL statement but accept the parameters. Suppose that the value of yearEnd is 2014, A2’s result will be as follows:

A3=create(row,col,value)

This line of code creates a table sequence with three fields – row, col and value – to store the transposed data and the summary values. The new table sequence is as follows:

Note: Similar to the database result set, a table sequence is also a structured two-dimensional table. But its genericity allows a field to have data of different types and its orderliness allows the data being accessed by their sequence numbers. These two features of table sequence are conveniently made use of in implementing this task.

A4: for ["visits","bookings","successfulbookings"]

Through accessing the set ["visits","bookings","successfulbookings"] by loop and appending data to A3’s table sequence, this line of code gets data ready for report creation. The working range of for statement, B4-C7, is represented by indentation instead of the parentheses or identifiers like begin and end. Within the working range, A4, the name of the cell where for statement resides, is used to reference the loop variable. During the first loop, for instance, A4’s value is “visits”.

Now let’s look at the code in the loop body.

B4=endValue=eval("A2(1)."+A4)    

This line of code dynamically retrieves order status data of the first record from A2. eval function can parse the string into an expression. For instance "A2(1)."+A4 will be parsed into A2(1).visits during the first loop and its result is 500. “A2(1)” represents the first record and “.visits” means retrieving the record’s visits field (as shown by the red box in the following figure).

C4=beginValue=eval("A2(2)."+A4)

Similar to endValue, beginValue dynamically retrieves order status data of the second record from A2. Its value during the first loop is 400.

B5=A3.insert(0,A4,A2(1).year,endValue)

C5=A3.insert(0,A4,A2(2).year,beginValue)

These two lines code insert records into A3’s table sequence. insert function is used to insert one or more records into a table sequence. Its first parameter specifies the position where the insertion happens. If the value of this parameter is 0, then append the record in the end.

During the first loop, for instance, B5 inserts “visits”, 2014 and 500 into the table sequence and C5 inserts “visits”, 2013 and 400 into it. Then A3 becomes this:

B6=endValue/beginValue-1

This line of code computes the growth rate of the specified year. Its value is B6=500/400-1=0.25 for the first loop.

C6=if(B6>0:"+",B6<0:"-")+string(B6,"#%")

This line of code is used to format the result of B6. The way is to add “+” before the percentage if B6>0 and to add “-” before it if B6<0. C6’s value during the first loop is “+25%”. Note: This step is not indispensable as data formatting can be executed more conveniently by the reporting tool.

B7=A3.insert(0,A4,string(yearEnd)+"/"+string(yearBegin),C6)

This line of code appends new records, such as “visits”, “2014/2013”, “+25%” during the first loop, to A3’s table sequence, as shown below:

Note that the type of these data is string, which is different from that of data previously inserted.

After the whole loop is executed, all data the report requires will have been appended to A3, as shown below:

result A3

This line of code returns the result table sequence in A3 to the reporting tool. esProc provides JDBC interface for integrating with the reporting tool that will identify esProc as a database. See related documents for the integration solution. 

Then a simple crosstab will be created with JasperReport, for instance. The template is as follows:

Define parameter pyearEnd in the report to correspond to its counterpart in the esProc script. The following is the preview of the final report:

The reporting tool calls the esProc script in the same way as that in which it calls the stored procedure. Save the esProc script as, say booking.dfx, to be called by booking $P{pendYear} in JasperReport’s SQL designer. 

esProc Simplifies SQL-style Computations – Transpose Rows and Columns

During database application development, we often need to deal with complicated SQL-style computations. The transposition of rows and columns is one of them. Oracle uses pivot function to realize this computation. The other databases haven't the counterparts to realize it directly, which makes the corresponding code difficult to write, understand and maintain. Besides, even the pivot function can only transpose the fixed columns, but is powerless about the unfixed ones. So are the other databases. Generally all of them must resort to the high level programming languages to realize the dynamic SQL.

However, coding this computation with esProc will be concise and easy to understand. We'll use an an example to illustrate this.

The following figure shows part of the SALES - a database table where order data are stored.

It is required to compute the total order amount, the maximum and minimum order amount, and the total number of orders of each month of the year 2013, and then transpose the data into a table with thirteen columns and four rows, in which the four operations occupy the first column, with subtotal being the column name, and every month covers a column, with the column names being 1, 2, 3, 4… The first five fields are as follows:

esProc code:

A1：Execute the SQL statement of selecting the data of the year 2013 and grouping and summarizing the data by the month. Result is as follows:

This simple SQL statement for data grouping and summarizing is supported by any database. The difficulty is the transposition of rows and columns following it.

A2=create(subtotal).record(["OSum","OMAX","OMIN","OCount"])

This line of code creates an empty table sequence where there is only one field: subtotal, as shown below:

Note: A table sequence is a data type in esProc. It is a structured two-dimension table similar to the data table of in SQL, but with more powerful function and more flexible usage. By the way, the result of A1 is a table sequence as well.

B2=A2.derive(${to(A1.len()).string()}).

This line of code adds twelve columns to the table sequence in A2 and thus forms the data structure after the transposition, as shown below:

derive function is used to add new columns to an existing table sequence so as to form a new one. For example, derive(1) means adding one column, where 1 is the field name and the field value is the same as the column name. derive(0:field1, null:field2) means adding two columns, where, respectively, field names are field1 and field2 and field values are 0 and null.

According to the requirement of transposition, twelve columns should be added here, for which the code should be derive(1,2,3,4,5,6,7,8,9,10,11,12). A macro, that is ${}, whose role is to convert a string into an expression, is used here in order to generate the code dynamically. to(A1.len()) in the macro is a sequence, whose value is [1,2,3,4,5,6,7,8,9,10,11,12]. The function string() is used to convert the sequence into the string "1,2,3,4,5,6,7,8,9,10,11,12".

A3-A5：Perform loop on A1, accessing one record each time, rearranging it vertically and, at the same time, modifying the corresponding column in the table sequence in B2. Please note the working range of the loop statement can be represented by the indentation, with no need of using braces ({}), or begin/end. So both B4 and B5 are in the working range and neither A4 nor A5 is in it.

Note: In esProc's loop body, the loop variable is the cell where for statement is entered. In other word, A3 can be used to reference the current record and A3.MONTH can be used to reference the MONTH field of the current record.

B4=A3.OSum | A3.OMAX | A3.OMIN | A3.OCount

This line of code concatenates the summarized fields of the current record in columns. The operator "|" represents concatenation. For example, the records of December in A1 should be like this after being concatenated:

A3.OSum in the code represents the OSum field of the current record. Since OSum is the second field of the records, it can be referenced by its sequence number, thus the code shall be written as A3.#2. Equally, the above line of code can be put as B4=A3.#2 | A3.#3 | A3.#4 | A3.#5.

B5=eval("B2.run(B4(#):#"+ string(#A3+1)+ ")")

This line of code means modifying the fields in B2 based on the result of B4.

eval function parses strings into expressions dynamically. For example, the computed result of eval("2+3") is 5; and here B2.run(B4(#): #13), the loop code for December, in eval function means inserting members of B4 in order into the 13^th column (i.e. December) in B2 according to the sequence numbers of the records in B2.

run function is used to modify the fields. For instance, run(field1+field2:field1, 0:#2) means modifying the value of field1 into field1+field2 and the value of the second field (i.e. #2) into 0.

#A3 means the current loop number. Its value is 1 when the first loop is being executed, and the value is 2 when the second one is being executed, and so on and so forth.

When the loop statement in A3-B5 is executed, the final result will be got in B2. The first several columns are as follows:

In addition, esProc program can be called by the reporting tool or a Java program in a way similar to that in which a Java program calls an ordinary database. The JDBC provided by esProc can be used to return a computed result of the form of ResultSet to the Java main program. For more details, please refer to the related documents.