Synch between QuickBooks and QQube is slow | CLEARIFY

Data Extraction Speed Issues

Extracting data on large files takes longer than expected, or appears to freeze.  Refreshing updated data using the manual refresh option takes longer than several minutes. 

Cause

There is no direct connection to the database, and all developers (including raw connectors) must use the Software Development Kit (SDK) - built in 2001 - to extract data.

It is a slow and kludgy text based extraction, and not a database connection extraction.  It will show a high CPU utilization, but normal memory usage.

But, it never lies, and is extremely secure.

The SDK was designed for putting data into QuickBooks - not extracting data out of it.

To further this conundrum, QQube extracts more data details from QuickBooks than any other developer in the Intuit Community.

EXTRACTION TYPES

There are several types of extractions:

• Detail Data Models.  Which contain all of the transaction line item details
• Summary Data Models. Which are aggregated by monthly, 52/53 or daily periods.

DETAIL DATA MODELS

• If you load your file from the very first transaction entry. It can take anywhere from 10 to 15 minutes on a file less than 100 Meg  - to a few hours for a very detailed file of 1 Gig.  
• Update refreshes generally take only a couple of minutes per QB company file.
• The following data models contain details at the item level:
  • Sales, Open Sales Orders, Accounts Receivable
  • Job Cost Details
  • Inventory
  • Purchases, Open Purchase Orders, Accounts Payable
  • Payroll
  • General Ledger Detail, Profit and Loss Detail, Cash Position

SUMMARY DATA MODELS

• Financial Summary Analytics and QuickBooks Enterprise Only Summary Analytics are reloaded each time, BUT don't take long to extract or refresh.  There are two exceptions to this: 
  • Balance Sheet by Class when there are large numbers of classes (e.g. 50 to 300) because QuickBooks calculates that report on the fly, and we are pulling that same "report" from QuickBooks, one class at a time.
  • Legacy (e.g. versions 6 and 7) Profit and Loss by Job (and by Class)  NOTE: Version 10 fixes this.

If it appears that your machine is freezing, that is never the case. It is expected behavior on larger files with the Intuit Software Development Kit (SDK)

Resolution

1. Only choose the data models you need. One model is not dependent upon another.  For instance, if you want job costing, just load job costing, NOT sales, purchases, general ledger, etc. as that information is already included in the job costing data model. (See Selecting QuickBooks Data Models in QQube)
2. Initial Data Loads should occur overnight for files over 750 Meg.
3. The faster the processor, the better the performance.  Time is money.  Don't put QQube on the "hand me down' machine.
4. Be sure to allow for 3 to 8 gigs of memory on hosted environments.

Dealing with Large QuickBooks Files

If your file is one of the following:

• QuickBooks Enterprise file approaching 1.25 Gig
• QuickBooks Premier files larger approaching 350 Meg
• QuickBooks Pro files approaching 250 Meg
• QuickBooks file containing more than 10 years of data

Then we strongly advocate getting someone like Matt Clark at https://www.qbornotqb.com to split your file into two files:  (1) with old data, that you can open - or store in QQube, and (2) a current file for every day use.  DISCLAIMER: We do not receive kickbacks from this company - we recommend him because he has successfully taken care of hundreds of customers over the last decade.