Huge array, out of memory

[jodleren]

Hi all

I am working on reading an intire tree into an array as this one:

TFileInfo = packed record
sFilename: string;
dtFileTime: TDateTime; // take up less memory than SystemFileTime
fFileSize: integer;
bFolder: boolean;
bChecked: boolean;
end;
TAFileInfo = array of TFileInfo;

But, when I have 3700 entries, I get an out of memory error.
I have done what I can to save memory, but I still lack it...
The record is packed, and the files does not hold their folders,
instead, the folder name is stored here, and I read it out on the fly,
and it applies for the following files.

But, how can I read in more than 3700 entries?

WBR
Sonnich

[Maarten Wiltink]

Post by jodleren
TFileInfo = packed record
sFilename: string;
dtFileTime: TDateTime; // take up less memory than SystemFileTime
fFileSize: integer;
bFolder: boolean;
bChecked: boolean;
end;
TAFileInfo = array of TFileInfo;
But, when I have 3700 entries, I get an out of memory error.

You mean SetLength fails? Or setting sFilename fails?

Post by jodleren
I have done what I can to save memory, but I still lack it...
The record is packed, and the files does not hold their folders,
instead, the folder name is stored here, and I read it out on the fly,
and it applies for the following files.

_That_ is actually rather a neat trick - providing you never need to
add any files later.

Post by jodleren
But, how can I read in more than 3700 entries?

I have no idea what the problem is, but I would never do this anyway.
Last time, I made a database. Before that, I constructed an in-memory
XML document. Today, I might write a few TCollection/Item classes (not
that different from the XML document, really).

Clearly you're quite desperate. Packing the record and switching from
SystemFileTime to TDateTime is all way out on the fringe. If you want
to save storage, don't declare a lot of Booleans but use a set that
xFolder and xChecked are in or not. That way you can actually pack eight
Booleans into a byte.

But you probably have a memory leak somewhere you didn't notice. Are
you calling FindClose?

It's hard to say anything useful without seeing more code.

Groetjes,
Maarten Wiltink

[ap]

Post by jodleren
Hi all
TFileInfo = packed record
sFilename: string;
dtFileTime: TDateTime; // take up less memory than SystemFileTime
fFileSize: integer;
bFolder: boolean;
bChecked: boolean;
end;
TAFileInfo = array of TFileInfo;
But, when I have 3700 entries, I get an out of memory error.

I don't think you can use Strings that way with Packet Records.

You maybe should have something like
sFilename: string[100];

..just a recollection, I did not test anything.
-ap

[Maarten Wiltink]

[...]

Post by ap

Post by jodleren
TFileInfo = packed record
sFilename: string;
dtFileTime: TDateTime; // take up less memory than SystemFileTime
fFileSize: integer;
bFolder: boolean;
bChecked: boolean;
end;
TAFileInfo = array of TFileInfo;
But, when I have 3700 entries, I get an out of memory error.

I don't think you can use Strings that way with Packet Records.
You maybe should have something like
sFilename: string[100];

_Usually_, you *should* not use long strings in packed records,
because the usual reason to pack a record is that you want to
write it to a file. But here, the record was only packed to save
memory.

Groetjes,
Maarten Wiltink

[Rudy Velthuis]

Post by ap

Post by jodleren
Hi all
TFileInfo = packed record
sFilename: string;
dtFileTime: TDateTime; // take up less memory than
SystemFileTime fFileSize: integer;
bFolder: boolean;
bChecked: boolean;
end;
TAFileInfo = array of TFileInfo;
But, when I have 3700 entries, I get an out of memory error.

I don't think you can use Strings that way with Packet Records.

Why not? As long as the string fields are not saved to file directly,
there is nothing wrong with it.

[Hans-Peter Diettrich]

Post by jodleren
TFileInfo = packed record
sFilename: string;
dtFileTime: TDateTime; // take up less memory than SystemFileTime
fFileSize: integer;
bFolder: boolean;
bChecked: boolean;
end;
TAFileInfo = array of TFileInfo;
But, when I have 3700 entries, I get an out of memory error.
I have done what I can to save memory, but I still lack it...

Incredible. I calculate about 64 KB for the array, maybe 100-200 KB
together with the filenames. If your application is out of memory with
that tiny amount of bytes, your application is at a dead end :-(

DoDi

[Jamie]

Post by jodleren
Hi all
TFileInfo = packed record
sFilename: string;
dtFileTime: TDateTime; // take up less memory than SystemFileTime
fFileSize: integer;
bFolder: boolean;
bChecked: boolean;
end;
TAFileInfo = array of TFileInfo;
But, when I have 3700 entries, I get an out of memory error.
I have done what I can to save memory, but I still lack it...
The record is packed, and the files does not hold their folders,
instead, the folder name is stored here, and I read it out on the fly,
and it applies for the following files.
But, how can I read in more than 3700 entries?
WBR
Sonnich

Use short strings for your sFileName field..

sFileNme:String[Max_Path];

etc..

[a***@aol.com]

On 21 May, 23:45, Jamie

Post by Jamie
Use short strings for your sFileName field..
sFileNme:String[Max_Path];

. . . or LongShort Strings.

ie write the string length as a 4-byte integer, followed by the
characters.

You need a couple of short functions to do that with streams, one to
write the length & then characters, & another to read the length, set
the string length & then read the characters into the string.

Works fine for me, for records or for TCollection/TCollectionItems.

Alan Lloyd

[Rudy Velthuis]

Post by jodleren
Hi all
TFileInfo = packed record
sFilename: string;
dtFileTime: TDateTime; // take up less memory than SystemFileTime
fFileSize: integer;
bFolder: boolean;
bChecked: boolean;
end;
TAFileInfo = array of TFileInfo;
But, when I have 3700 entries, I get an out of memory error.
I have done what I can to save memory, but I still lack it...

Show your code. How do you allocate the array?

Oh, and FWIW, Integer is far too small for a file size, unless you can
be sure all your files are that small.

[Jamie]

Post by Rudy Velthuis

Post by jodleren
Hi all
TFileInfo = packed record
sFilename: string;
dtFileTime: TDateTime; // take up less memory than SystemFileTime
fFileSize: integer;
bFolder: boolean;
bChecked: boolean;
end;
TAFileInfo = array of TFileInfo;
But, when I have 3700 entries, I get an out of memory error.
I have done what I can to save memory, but I still lack it...

Show your code. How do you allocate the array?
Oh, and FWIW, Integer is far too small for a file size, unless you can
be sure all your files are that small.

Interesting, what are they using D1 ? I say that because the math
comes out to exceed a 64k page.

Or, its part of a massive app with lots of memory frags around.

Actually, I think if one was to set the array size prior too, it may
work out, instead of allowing D to do it on the fly.

Maybe the error is miss leading and D has an issue with the amount of
system dynamic strings being allocated at once..

Lets see, the system must allocate space per record as you go and
allocate space for that string, which is done there after. I suggest
that maybe these records are getting populated on the fly and with the
added string content the system maybe having issues with frags!..

Just an assumption. Don't pay attention to me, i'm just a guy from
Maine, where everyone is related..

Jamie.

[Rudy Velthuis]

Post by Jamie
Maybe the error is miss leading and D has an issue with the amount of
system dynamic strings being allocated at once.

I doubt it.

[a***@aol.com]

Do some debugging, put a listbox on your form and display the record
count, AllocMemCount & AllocMemsize during your treenode allocation to
records (maybe every 1000 records). See if the memory used agrees with
the estimates, and it do not, find out why.

Perhaps your diagnosis of the trigger for the error message is
incorrect.

Alan Lloyd