Some of the reports in my Delphi VCL app are anized around TClientDataSet. There is one main CDS (which is the backbone of the report and displayed in the grid) and there are some queries. There is one main query which retrieves data and posts it to CDS:

MainQuery.First;
while not MainQuery.Eof do begin
  CDS.Append;
  CDSF_1.AsString:=MainQueryF_1.AsString;
  CDSF_2.AsFloat:=MainQueryF_2.AsFloat;
  ...
  CDS.Post;
end;

After the data is loaded inside CDS (or during the loading), my program can make some additional computation.

Of course, I am doing everything to make the described load as fast as possible:

• CDS.DisableControls
• CDS.IndexFieldNames='', CDS.IndexDefs.Clear
• CDS.LogChanges:=False
• CDS.AggregatedActive:=False; In fact, I completely removed any aggregated field from CDS
• CDS.AfterScroll - no such event
• CDS.BeforePost/AfterPost - minimalistic, fixed default values only, on field OnChange

I can not imagine anything else that can boost optimization.

My CDS has 60 fields - minimalistic - no blobs, no nested tables, no master-details relationships.

Yet still - I am trying to insert 20k records (which is not that much) and the first records are inserted into CDS at 100 inserts per second, but it slows down. After insert of 3k records, the rate has dropped to 5 inserts per second, and the rate drops still further to 1 insert per second, and so on.

The work with TClientDataSet is not possible.

I have used TClientDataSet from time to time, yet such behavior is shocking to me. Delphi DataSets usually can handle even 100k+ records without much problem.

Are there any known limits to TClientDataSet in terms of RecordCount? Or maybe I am omitting some very important optimization?

I can not use TDataSetProvider because I am fetching data in multiple chunks. I tried to fill an OleVariant typed array:

MainRecordArray := VarArrayCreate([0, 15999], varVariant);
...
//loop over MainQuery
RecordData := VarArrayCreate([0, CDataSet.FieldCount - 1], varVariant);
MainRecordArray[MainRecordIndex] := RecordData;
Inc(MainRecordIndex);

And then:

CDS.AppendData(MainRecordArray, False);

But this raises an error:

Trying to append data to a non-partial dataset.

As I understand, AppendData() is something from internals and there can be no good explicit uses cases.

So, am I hitting the limits of TClientDataSet, or am I missing some optimizations?

To be honest, I am doing my programming in Delphi 6 (so, 25 years old) and Delphi 2009 (15 years old) with midas.cds from around 2015. But, it may be possible that these limits are in the more recent Delphi as well?

Note added. I have also tried to run my program with midas.dll 28.0.42600.6491 from the (almost) most recent Delphi and there was error message Error loading MIDAS.DLL. and this is happening both for Delphi 5 and Delphi 2009 programs. So, no use of the most recent midas.DLL for the older Delphi programs. Of course, I did all the required choreography around midas.dll - I unregistered the all midas.dll, overwrote it with more recent midas.dll in C:\Windows\SysWOW64 and registered it with regsvr32.

Note 2 added. The same error message Error loading MIDAS.DLL. is in the case when there is no midas.dll or it is not registered. So - maybe (more recent) midas.dll is required some additional libraries and just copying and registerint it alon on the client computer is not enought?