We have a large QBW file at 1.6 GB which we are addressing shortly but I have noticed that even with this file size the initial querying of the QuickBooks file by the Manual Refresh in QQube is fairly quick. The part of the process, for us at least, that takes the longest is the "Refreshing #####" where the ##### is perhaps a table number like 20826 or similar, after the querying is completed. It can take half an hour or more. When I look at the Windows Task Manager while this part of the Manual Refresh is running I can see that this process is only utilizing one out of the 12 available cpu cores (logical processors actually). I realize that some workloads are just plain single threaded and can't be changed but I'm curious if there a setting to enable use of all of my computer's cpu cores to speed up this process. I apologize in advance if this is a stupid question! :)
Regards,
Rick
I would have to look at the log to give you an educated answer. But here are some generalities for others to peruse, if they view this post.
First I would consider splitting that file up into old and new at someone like Matt Clark www.qbornotqb.com especially if you have many years of data. This is more about your QB file than QQube. After 1.25 Gig (despite what Intuit tells you), we tell people that their file is a heart attack waiting to happen - and there is truth to that. Plus it slows everybody down, and corruption is inevitable - and that is regardless of what the verify and rebuild say; they are not related to the Intuit SDK in any shape or mannerism.
If you are loading detail data like job costing or general ledger or inventory, it would take many hours - probably half a day to load all of the data the first time. Load and updates should be pretty quick, including the refresh. If you are loading detail data the first time, and it is quick (except for financial summary), I am suspicious that a previous synch had errors, and the data was not loaded from scratch after a QB file rebuild.
If you are loading summary data like financial summaries, then the initial or updated load and refresh times should all be quick.
There are several exceptions to this:
Audit trail and sales tax are just incredibly difficult to transform because of the difficulty of exposing that data via the Intuit SDK. Just can't be helped.
The Profit and Loss by Job and by Class - especially with a lot of jobs, classes, and accounts - takes quite a bit of time to refresh because it is a cross join of those three items.
Lastly, not everything needs to be loaded as one analytic does not depend upon one another; each one is complete. Most people load everything - because they want to play with all of the features - not realizing that it is hardly ever necessary; especially after you have explored the tool and data. See this guide on Selecting Data Models: Data Model Selections
Since developers don't have direct access to the data, we have to perform gyrations to balance the time it takes for data transformation for measures that can be used out of the box, with a usable experience.
From the feedback we receive, the initial load is not bothersome. The subsequent refreshes - would; and you have indicated that (although your quick initial is suspicious..., until we see the log)
You can also turn on verbose logging that will show exactly what is being refreshed - just for your own reference. But it still requires our eyes for specificity as to the cause.
To see the actual table names instead of table numbers, we have the following Easter Egg:
I would highlight the last log, and hit the Send to CLEARIFY button, so we can see the specifics of your file, what you are pulling, and what is taking time. We can pinpoint everything from that log - and is always our first place to start. Every QB file is different, as is the data extraction needs.
Thank you for your great question.
Choose a location
Forum :
Your session has expired. You are being logged out.