Automatic Regression Testing of Queries
With every release upgrade of a SAP BI system there is a high risk that some queries will not work anymore, or – even worse – deliver wrong results. Particularly queries using special features likes special aggregation, replacement path formula variables, unit conversion etc. have a higher chance of slightly changing their behavior with an upgrade.
Manual spot checks will typically not find these bugs during the upgrade. Your odds can be increased however, if you test a very high number of queries / selection variants. Having in total more than twenty thousand user variants of more than thousand queries at a customer was a good basis for automatic regression testing with a quite good coverage.
I have developed on the ABAP side programs that will precalculate saved selection variants of queries and save them as PDF files (using the BEX Broadcaster), and developed a .NET program that can parse and compare the PDF files to see if they exactly match in content. Of course you need to run this precalculation before the and after the upgrade and make sure not to change any data (incofube and master data) in between. The test application will can show the difference between old and new report both in text format as a ”diff”, or overlay the two PDFs graphically and highlight the changed spots.
During every upgrade we could unveil in average 3-4 systematic changes in query behavior of whic normally 1-2 were critical (e.g. wrong data, or different behaviour of NODIM function) and required a fix, while others were just of cosmetic nature (e.g. irrelevant change in sorting order).