Update: 2011-12-07 : SAM Reports 2 has been launched, as you may see on this blog post.
Accessing sliced log files
I’ve been working on a new version of SamReports for quite some time. My main goal was to provide seamless “integration” with Trixbox/Fonality SKU-s. (It’s not integration really, but the process of copying thousands of files from PBX, gluing them together, processing and so on, has to be transparent to the user, with as little hassle as possible, none preferably).
I came into possession of a large dataset and that has helped a lot. You cannot possibly simulate real, raw data to work with. You would just end up with gibberish. That brings the issue of processing..
Current version of SamReports tries to load all the data in memory, to generate reports. That just doesn’t work for millions of log lines.
The new version is capable of processing any amount of data, because it works in chunks, and generates reports sequentially. I tested on a dataset with more than 10 million lines in CDR, and it finished in 37 minutes.
Above is the image of a summary generated after processing the log files. The graphs are being drawn during the parsing so that you’re not bored while processing a large dataset for the first time This is displaying the same charts to get an overview.
Quirks and twists of master.csv (chopped or not)
I did get dirty with the local and transferred calls in CDR logs. Sometimes a call can be spanned in 5, 6 log lines. Those lines don’t necessarily come close, but can be even 1000 lines apart.
Here you see a call spanned into just 2 lines, one after the other.
To be continued…
This is just a preview of what’s been cooking. There are many other things to mention, but I will do it in another post…