Developers

The Report Process

By David Swain
Polymath Business Systems

In the last article we surveyed some of the basic features of an Omnis Studio Report Class, focusing on the layout side of the Class and comparing and contrasting Report Classes with Window Classes. This time we will detail the process that Omnis Studio follows when it generates a report automatically from a native Omnis database with no programming intervention. We do this not because the native Omnis database is "superior" in some way, but because this process makes a good model for us to follow when building our own reports where we want to take more control of things. In future articles, we will explore ways in which we can override or enhance certain of these built-in features.

An "automatic" report in Omnis Studio is a "black box" in many ways. The technologies used within the process are not exposed to us, so we must do a certain amount of "educated guessing" to understand what happens between execution of the Print report command and our first glimpse of the finished report. What follows is my interpretation of the process...

Going With The Flow

Here is the basic flow of the Omnis Studio report process. For maximum use of features, let us assume that we are running a complex report that includes the use of a search to limit records, a number of sort levels, subtotaling on some of these levels to generate aggregate values for multiple variables within the report, totals at the end of the report and a header and a footer on each page. We will detail many of the pieces used to manage all of this information later in the article.

First, Omnis Studio goes through a setup phase. There are a number of memory partitions that must be established to handle sub-processes from record sorting to page layout. While no such constructs are mentioned in the Omnis Studio documentation, a thoughtful review of the report process indicates that such things must be created - so we will give them names to aid in discussing these inner workings.

Next, Omnis Studio must determine which records will be included in the report and in what order they are to be presented.

After this has been done, Omnis Studio proceeds to process these records one at a time, adding a Record Section to the current page image. At the same time, aggregate values are accumulated for use in subtotal sections. In fact, Omnis Studio must check whether one or more of the reports Subtotal Sections needs to be added to the current page as well as checking to see whether it is time to begin a new page.

As each page is filled, Omnis Studio closes the page and releases it to the current printing device. That is, of course, unless some special feature in the report requires that the filled page be held for later update as more pages are added. We will consider one such feature here.

After the last record has been processed, Omnis Studio places the last group of Subtotal Sections onto the current page. It then does the same thing with the Totals section.

When the End of Report Section is encountered, Omnis Studio releases any remaining pages to the printing device (after performing whatever action any earlier pages require before release). It then performs various housekeeping chores, releasing any memory areas it had used during the report process.

This is the short version. Now let's go into a little more detail about some of these mysterious stages...

Blind Man's Buffers

The first thing Omnis Studio must do when called upon to print a report is to set up certain workspaces in the computer's RAM. The technical term for such a workspace is a buffer. Each buffer serves a different specific purpose in the overall process. While these buffers are not mentioned in the Omnis Studio documentation, we know that they must exist because of the work that must be done to organize a report. So let us give names to them and then consider how they may be structured and manipulated. This exercise will help us understand how to emulate these features should we ever need to augment or override the Omnis report process with our own code.

The first of these buffers that Omnis Studio must allocate is what we will call the Sort Buffer. When we set up the Sort Fields window for multiple levels of sorting within a report, we are specifying the structure of this buffer. As part of the initialization of a report, Omnis Studio uses the variable names we provide in the Sort Field information to define the Sort Buffer. This is essentially a list in RAM used to hold images of the records for a report and to put those images into the proper order. Certainly the Omnis datafile can contain indexes for maintaining scanning order within the database, but each index links a single value with a record. True, that value may be a composite from multiple columns, but it is still a single value and does not offer the flexibility of the nine sort levels of a report.

The Sort Buffer can have up to nine columns of ordering values and each column can be independently sorted in the ascending or descending direction. In addition to these columns is a column for the Record Sequencing Number (RSN) for each record. Whether or not a Sequence variable is included in a File Class, each record in that File still has a unique RSN value that Omnis Studio uses to uniquely identify it. This is the most important column in the Sort Buffer, because the purpose of the Sort Buffer is to get the Main File RSN values into the proper order for the printing of the report.

Another buffer that Omnis Studio must set up in the initialization phase is what we will call the Subtotal or Aggregation Buffer. This is a structured collection of variables used to accumulate count, total, minimum and maximum values for specific variables at each of nine potential subtotal levels as the report proceeds. We can also retrieve an average value at a subtotal break, but I suspect that this is derived from the count and total values rather than maintained separately (but then, I don't really know this for certain - that's just how I would have done it). Other statistics, such as the median value, can also be determined from these basic building blocks. (If we want to also track the sum of squares for determining standard deviation, etc., we must manage such values "manually" - but that is a subject for another article...)

During the initialization phase of a report, Omnis Studio seems to examine the Subtotal and Totals sections of the Report Class to determine which variables are represented by fields with a totalmode setting other than kTMNone. It then adds a slot for that variable in the Aggregation Buffer. Omnis Studio appears to track all of these cumulative values for each such variable. If only a field for kTMTotal is given, the $count, $minimum, $maximum and $average values are also available through Notation. But if no aggregate fields occur for a variable, its notational aggregate properties remain empty.

A third buffer that Omnis Studio sets up at this point is a page layout buffer we will call the Virtual Page. Omnis Studio does not send each element of a report directly to the printing device. Rather, it lays out an enitre page and then sends the completed page description to the printing device. So a memory buffer must be needed for the accumulation of this information. Under some circumstances, such as when the Page Count object is used in a report, Omnis Studio must hold nearly-completed pages in memory and only release them when they can be finalized. In the case of the Page Count object, the entire report must be held in RAM until it is completed so that the total number of pages can be communicated back to each page.

Once these buffers have been established, Omnis Studio can then move on to the next phase of the report.

Gather Ye Records...

That next phase is the gathering of record images for sorting purposes. Omnis Studio must scan the Main File for the report to populate the Sort Buffer. If a search is invoked along with the report, then only those records that meet the search criteria are represented in the Sort Buffer. Only the values of variables declared in the Sort Fields window, plus the Main File RSN, are gathered at this time. If a variable was specified as Upper case in the Sort Fields window, I suspect that the values for that variable are converted to upper case on addition to the Sort Buffer.

Once all the record images have been read into the Sort Buffer, the list is then sorted on the criteria (ascending or descending) given in the Sort Fields window. The record images held there are now in the proper order for the report, so Omnis Studio goes to the top of this list and reads those records a second time, but this time the entire record is used...

A Matter Of Record Sections

The processing of records through the Record Section is a pivotal part of the report process. After sorting, each record is presented to the Record Section. A number of things happen while the Record Section is in charge. Each field is populated in its turn with either the value of its associated variable or by evaluating its calculation expression. The layout description of each field is then positioned within a Virtual Record Section that will be positioned on the Virtual Page in its turn. But this is just the first step...

A Place For Everything...

As each Record Section is filled, it is added to the Virtual Page. The layout of record sections on the Virtual Page continues until the current page is filled. In the simplest scenario, this page is then closed and released to the printing device. If the report is a label report, then the Record Section images are placed in the appropriate position within the label grid, otherwise they are simply stacked one below the previous one. The property values assigned to the Report Class ($islabel, $labelcount, $labelwidth, etc.) determine how the Record Section images are arranged.

Record Sections are placed one after the other on the same page unless a page break is called for. In the automatic reports that we are following in this article, this occurs either when a section does not fit completely onto the current page (in which case, that section is split between the current and the next page), when the pagemode property value of the section requires it (for kNewpage, kTestspace or kFitonpage values of this property) or when a Sort Field requires it. Before placing the Record Section on the Virtual Page, Omnis Studio consults the Sort Buffer to determine whether the value of any sort field with a newpage property value set to kTrue has changed sufficiently from the previous record to require a page break. So the Sort Buffer's job is not fulfilled by simply sorting record images - it continues to take an active role in the processing of records. (We will see that it has yet another job a little further in the process.)

As new pages are required, Omnis Studio first lays out the page border, background picture and pattern, etc. and the Page Header Section for that page. It also reserves space for the Page Footer Section. The Footer is not laid out until the page is otherwise completed and about to be released to the printing device. In an automatic report the Header and Footer will remain the same size from page to page. In a future article we may explore how we can deal with these items more dynamically.

If the repeat factor for the report is greater than one, multiple copies of this image are placed on the Virtual Page. If the repeat factor is zero, the Record Section image is not added to the Virtual Page. This creates what we will call a summary report - one with only Subtotal and Totals Sections. But the rest of the Record Section processing does occur whether or not the Record Section image is mounted on the Virtual Page...

The Bucket Brigade

While Omnis Studio processes each Record Section, it also determines which variable values (which can be either File Class or Instance variables) must be passed to the Aggregation Buffer. Each such value is then accumulated into each of the associated "buckets" for the various Subtotal levels and tested against the current minimum and maximum values for each Subtotal level. The count bucket for each Subtotal level is also incremented by one. In a complex report, this is a lot of "take one down and pass it around" processing for our nine bottles of subtotal accumulation (plus one for the Grand Total - and times four or five for the various types of aggregation)!

When a Subtotal Section at a specific level is triggered, these values are then available for use by fields with non-zero totalmode values in that Section and for notation strings that access the various aggregate values of the variables represented in the Aggregation Buffer. After that Subtotal Section is completed, the buckets for all the values at the Subtotal Level (and any lower levels) are emptied to begin accumulating anew for the next subtotal grouping at the level.

But just how does Omnis Studio know when to build a Subtotal Section?

Clock Watchers Waiting To Take A Break

Before a Record Section image is placed on the Virtual Page, Omnis Studio tests each variable represented in the Sort Buffer against the value of the previous record to see whether the subtotal triggering criteria have been met for any Subtotal Level. Only Sort Fields with their subtotals property value set to kTrue need to be tested. If the criteria for one or more Subtotal Level breaks is met, those Subtotal Sections are processed in order from highest to lowest level number.

As the content of a Subtotal Section is being compiled, the Aggregation Buffer values for the corresponding Subtotal Level become available. This is the only time we have access to these values! They are not exposed to us in any way I have been able to discover while they are being accumulated. They are not even available from the $print method of that section! So if we need to retain them for use elsewhere in the report (such as building a list of subtotals for a graph or a summation list in our Totals Section), we need to use fields on the section layout to assign these values to other variables (and only hash variables seem to have the stability or endurance for this purpose in my testing so far...).

For labeling and review purposes in a Subtotal Section, Omnis Studio "rolls back" the values from the most recent Record Section by default during the processing of a Subtotal Section. This allows us to label items in the section like "Totals for ..." with appropriate values. Remember that a Subtotal Section is only triggered by the current record moving beyond the sort value on which the Subtotal Level depends. This rollback feature is extremely important in helping us create understandable reports. Of course, we can override this feature for specific variables by setting the noreload property value of a field representing that variable to kFalse, but we usually shouldn't bother...

End Of The Line

As Omnis Studio cycles through the Sort Buffer, it has no problem detecting when it has reached the last record for the report. When this occurs, a number of things must happen: The final collection of Subtotal Sections must be processed and added to the Virtual Page, the Totals Section must also be processed and added to the Virtual Page, the final page and all pending pages must be released to the printing device and the process buffers used during the report must be cleared and the memory they used deallocated.

These last two items are the responsibility of the End of Report Section. This section does not just signify the "bottom" of the report layout. It signals Omnis Studio that there is no more report processing to be done. This means that it is now time for the remainder of the report to be passed to the printing device and for the allocation of the report buffers to be reversed.

In older generations of Omnis, it was possible to not include an End of Report Section. People who did not understand the importance of this section simply failed to include it or decided to leave it off because it "didnt' do anything" in their view. These same people did notice, however, that when a second report was printed, the last page of the first report magically appeared in the printer. And occasionally, after the printing of a number of reports, their clients began getting "Out of memory" errors and had to restart Omnis. These were not "bugs", but simply the result of not using the End of Report Section to perform its janitorial duties.

In Omnis Studio, the Record and End of Report Sections are not optional (they never were, actually) or removable. They are the first objects automatically included in a new Report Class (ident numbers 1001 and 1002 respectively) and are mandatory for any report.

End Of Report Process Article Section...

I hope you found this review of the report process informative. I realize there are no code listings and no screen shots in this article, but we are just setting the ground work for future articles.

In the next article we will briefly look at list-based reports and then begin to examine how we can supplement the built-in features of Omnis Studio reports to deal with more complex tasks and data sets than Omnis can in "automatic" mode.