Discussion:
Delphi D4: using pointers to transfer data from COMM queue to circular queue
(too old to reply)
P E Schoen
2016-08-19 06:19:06 UTC
Permalink
I have an application that uses a serial port component to read and write
data through a USB serial port at 57.6 kb. Originally I used the SerialNG
component and with some tweaking I was able to get it to work reliably. But
it has problems with Win10, so I changed the code to use ComDrv32, which
works when used in a simple TTY demo.

The SerialNG component uses clusters or packets which can be set to a range
of values. I am communicating with a USB device set up as a CDC serial
emulator using Microchip PIC18F2550. It is basically a data acquisition unit
that samples at 2400 samples/sec with a 10 bit ADC. I am splitting the data
into two 5 bit values in 8 bit data characters, and I'm using the remaining
three to identify the data as a MSB or LSB (using one evem/odd bit), and a
two bit value to determine if the data has been received in proper sequence.

Here is the function that receives the data from the serial port component:

===============================================================
procedure TfmCommLib.CommPortDriver1ReceiveData(Sender: TObject;
DataPtr: Pointer; DataSize: Cardinal);
var
RecdPtr: PChar;
CSize, NCSize : Integer;
n: Integer;
begin
RecdPtr := PChar(DataPtr);
RxDcount := DataSize;
if RxDcount > MaxRxDcount then
MaxRxDcount := RxDcount;
if CommBufferPtr + DataSize >= MAXCOMM then begin
move( DataPtr^, CommBuffer[CommBufferPtr], MAXCOMM-CommBufferPtr );
n := DataSize - (MAXCOMM-CommBufferPtr);
RecdPtr := RecdPtr+n;
DataPtr := RecdPtr;
CommBufferPtr := 0;
move( DataPtr^, CommBuffer[CommBufferPtr], n );
end
else begin
move( DataPtr^, CommBuffer[CommBufferPtr], DataSize );
CommBufferPtr := CommBufferPtr + DataSize; end;
CommCharCount := CommCharCount + DataSize;
if(CommCharCount > MAXCOMM) then
CommCharCount := MAXCOMM+1;
end;
========================================================

It works, but I get errors after running for about 20 seconds. That
corresponds to the point where the CommBuffer will overflow the MAXCOMM
value of 100,000 (4800*20). My previous code for SerialNG used a cluster
size of 2 and a loop to transfer the received data to the buffer one byte at
a time. When I tried that, it seemed to work poorly. So I used a variation
of the method from the ComDrv32 TTY demo using move( DataPtr^,
CommBuffer[CommBufferPtr], DataSize ); But you can see that I had to split
the transfer when near the end of the queue.

I had to use a separate PChar pointer RecdPtr, and I also had to use an
integer for the pointer arithmetic. Apparently Delphi does not allow RecdPtr
:= RecdPtr + (DataSize - (MAXCOMM-CommBufferPtr) ); RecdPtr + n is OK. I'm
not sure if I could just advance the DataPtr in the same way, since it is
declared as a generic Pointer type.

I can't very well use the Delphi debugger to see what's happening, as the
data is received continuously and there is no handshaking.

TIA for any suggestions.

Paul
P E Schoen
2016-08-19 07:58:01 UTC
Permalink
I mentioned elsewhere that I used TCriticalSections previously during the
transfers from the ComDrv32 component input buffer to the larger CommBuffer,
and also when processing the CommBuffer to detect and compensate for errors.
This seemed to work for the SerialNG component, but for ComDrv32 it seemed
to increase the errors. Removing them, I got some errors starting at 20
seconds as before, but after 700 seconds there are only 146 errors compared
to nearly 1000 previously. There is probably a thread running to receive the
characters into the COM16 file.

I tried various settings for the serial port component with inconsistent
results. What is consistent is that errors always start when the CommBuffer
is nearly full and the data must be split to fill the buffer and then start
again at the beginning. I may need to check my arithmetic.

Paul
P E Schoen
2016-08-19 08:42:54 UTC
Permalink
I was able to rewrite the receive procedure and now it works with no errors.
I think I have used pointers properly here:

==========================================================
procedure TfmCommLib.CommPortDriver1ReceiveData(Sender: TObject;
DataPtr: Pointer; DataSize: Cardinal);

var
RecdPtr: PChar;
n: Integer;
begin
RecdPtr := PChar(DataPtr);
RxDcount := DataSize;
if RxDcount > MaxRxDcount then
MaxRxDcount := RxDcount;
For n:= 1 to RxDcount do begin //2
CommBuffer[CommBufferPtr] := RecdPtr^;
inc(RecdPtr);
inc(CommBufferPtr);
if(CommBufferPtr > MAXCOMM) then
CommBufferPtr := 0;
inc(CommCharCount);
if(CommCharCount > MAXCOMM) then
fmCommLib.CommCharCount := MAXCOMM+1;
end; //2
end;
========================================================

In the previous procedure for SerialNG, I did this:

var
RecdPtr: PChar;
begin
RecdPtr := CommSerialPort.ReadNextCluster( NCSize, RdErrors );
...
FreeMem( RecdPtr, NCSize ); // Added 4/8/09
end;
========================================================

I don't know if that was a problem or not. It seems that I might need to
reserve memory for the PChar before using it?

Paul
Hans-Peter Diettrich
2016-08-19 10:34:55 UTC
Permalink
Post by P E Schoen
var
RecdPtr: PChar;
begin
RecdPtr := CommSerialPort.ReadNextCluster( NCSize, RdErrors );
...
FreeMem( RecdPtr, NCSize ); // Added 4/8/09
end;
========================================================
I don't know if that was a problem or not. It seems that I might need to
reserve memory for the PChar before using it?
Various implementations of ReadNextCluster are possible. It may be
required or forbidden to free the memory of the returned pointer.

DoDi
Hans-Peter Diettrich
2016-08-19 10:30:04 UTC
Permalink
Post by P E Schoen
I tried various settings for the serial port component with inconsistent
results. What is consistent is that errors always start when the
CommBuffer is nearly full and the data must be split to fill the buffer
and then start again at the beginning. I may need to check my arithmetic.
Can't you use multiple buffers, to eliminate the moving of data?

DoDi
P E Schoen
2016-08-20 00:54:54 UTC
Permalink
Post by Hans-Peter Diettrich
Post by P E Schoen
I tried various settings for the serial port component with inconsistent
results. What is consistent is that errors always start when the
CommBuffer is nearly full and the data must be split to fill the buffer
and then start again at the beginning. I may need to check my arithmetic.
Can't you use multiple buffers, to eliminate the moving of data?
Much of this code dates back to around 1996-1999 when I first tried making
an application for Windows using Borland C++, and my first use of Borland
Delphi 4 seems to be around 2003, when a consultant for another project
showed me how to use it. I think I purchased Delphi and BC++ around 1998.

I'm not sure how multiple buffers would help. The serial port component has
its own inBuffer which I have set to 48,000 bytes, or enough for 10 seconds
of data. There is probably another small buffer at a lower level (there is
an FTempInBuffer which is a pointer to data space allotted to
FInBufferSize). The actual reading of data is accomplished by using:

nRead := 0;
nToRead := comStat.cbInQue;
if (nToRead > 0) and ReadFile( FHandle, FTempInBuffer^, nToRead, nRead,
nil ) then
if (nRead <> 0) and Assigned(FOnReceiveData) then
FOnReceiveData( Self, FTempInBuffer, nRead );

So it does not appear that the FTempInBuffer is a circular queue, and thus I
need to implement that in my own code using my CommBuffer[]. I could
probably use much smaller values for the buffer sizes, but that's not really
an issue these days when most computers have at least 2g of memory. The
RxDataCount displayed in my status dialog is usually 280-390 bytes, which
may depend on the PollingDelay which I have set at 50 mSec. At 4800 char/sec
that is 240 characters. The extra characters likely occur during the buffer
processing procedures.

I am doing additional processing in a procedure called by a timer set at
50-250 mSec. So that may account for more delays and accumulation of data in
the buffers. My MaxRxdCount after 10 minutes is now 1758. I have found that
other Windows processes can increase this number, but so far it does not
seem to lose data.

Quite a lot of real time processing occurs during these update intervals.
Character pairs are read from the CommBuffer circular queue, and are parsed
to verify their validity according to the extra two bits in each character,
while the remaining 6 bits in each are concatenated to form a 12 bit
integer. The ADC is only 10 bits but I designed the system to use 12 bits if
needed. The value is shifted by 2048 to get a signed integer.

The absolute value is compared to a threshold value and a number of samples
to determine if the value (of current) is enough to indicate that a test is
in progress. If so, a small circular queue is used for pre-trigger data and
added to a Waveform data buffer (like a digital storage scope). When the
value drops below threshold for a number of samples, the actual start and
stop points of the waveform are calculated, and a true RMS computation is
performed to give the value of current in the pulse. This may be repeated
for up to five pulses of 60 Hz current in a single test, with off times of
as long as 20 seconds (although usually 1 or 2).

Meanwhile, the true RMS value of the samples in each update interval is
computed and displayed on the screen, giving a continuous RMS value at all
times. The sampling intervals of 50, 100, 150, 200, and 250 mSec are chosen
to use an integral number of zero crossings for either 50 or 60 Hz. At 2400
samples per second, 50 mSec is 120 samples, and corresponds to 3 cycles at
60 Hz or 2.5 cycles at 50 Hz. This measuring technique is used in DMMs to
cancel the effects of line voltage noise. It also produces consistent true
RMS readings when the start and end of the sampling period are not
synchronized to the zero crossings.

You may have already "left the building" at this point, unless you find this
sort of discussion interesting. I have been involved in the measurement of
power line frequency current and voltage signals for a LONG time, close to
40 years. Technology has changed since then, from mostly analog methods,
then ADCs with early microprocessors like the 8085 and Z80, then signal
processing boards with MSDOS, and now PICs and DSPs with Windows 10 or other
OS.

Thanks for your valued suggestions and time.

Paul
Hans-Peter Diettrich
2016-08-20 10:43:43 UTC
Permalink
Post by P E Schoen
You may have already "left the building" at this point, unless you find
this sort of discussion interesting. I have been involved in the
measurement of power line frequency current and voltage signals for a
LONG time, close to 40 years. Technology has changed since then, from
mostly analog methods, then ADCs with early microprocessors like the
8085 and Z80, then signal processing boards with MSDOS, and now PICs and
DSPs with Windows 10 or other OS.
Currently I see similar (but hobbyist) projects in the Arduino forum. I
never tried real time programming on Win32/64, nor did I use COM port
components with Delphi. In the early PC days I did some data acquisition
using GW-Basic, later with BCB on Win3. I'd no more try to make my hands
dirty with real time programs for Windows, and in detail not with
Delphi, be some old reliable (up to D7) or uncomfortable and bloated new
(RAD Studio) versions. It's interesting to hear that you still are in
business :-)

DoDi
P E Schoen
2016-08-20 23:15:48 UTC
Permalink
Post by Hans-Peter Diettrich
Post by P E Schoen
You may have already "left the building" at this point, unless you find
this sort of discussion interesting. I have been involved in the
measurement of power line frequency current and voltage signals for a
LONG time, close to 40 years. Technology has changed since then, from
mostly analog methods, then ADCs with early microprocessors like the
8085 and Z80, then signal processing boards with MSDOS, and now PICs and
DSPs with Windows 10 or other OS.
Currently I see similar (but hobbyist) projects in the Arduino forum. I
never tried real time programming on Win32/64, nor did I use COM port
components with Delphi. In the early PC days I did some data acquisition
using GW-Basic, later with BCB on Win3. I'd no more try to make my hands
dirty with real time programs for Windows, and in detail not with Delphi,
be some old reliable (up to D7) or uncomfortable and bloated new (RAD
Studio) versions. It's interesting to hear that you still are in business
:-)
Yes, I'm still in business, although at age 67 I've been semi-retired for
about 2 years. I still support my Ortmaster products, including the original
ORTM-1 and ORTM-2 that use MSDOS and the LPT parallel port to interface to
the data acquisition device I designed around 1994 and produced until about
2004 when I designed the first ORTM-3 (serial interface) and later the
ORTM-4 (USB interface). Some customers still have and use the ORTM-2 but it
is becoming more difficult for them to find working computers with MSDOS (or
Win95) and native parallel ports. That's good for me, because I offer the
only viable upgrade option for modern Windows machines.

I have considered upgrading to a newer version of Delphi. I have heard that
the even numbered releases (D4 and D6) were buggy and their odd-numbered
successors (D5 and D7) were much better. I found some Delphi Enterprise 8
products on eBay for under $200 and CodeGear Delphi 2007 Enterprise for
$430. Embarcadero does not offer an upgrade deal from D4, and their lowest
level (Pro) version that offers database support is $1405.
https://www.embarcadero.com/app-development-tools-store/delphi

Perhaps even better is the open source Free Pascal and the Lazarus IDE that
has a look and feel very similar to the Borland Delphi IDE that I am used to
with D4-Pro.
http://wiki.freepascal.org/Lazarus_Documentation
https://www.youtube.com/playlist?list=PLB24C56953A79987A

Thanks for the information and discussion.

Paul
Hans-Peter Diettrich
2016-08-21 10:50:38 UTC
Permalink
Post by P E Schoen
I have considered upgrading to a newer version of Delphi. I have heard
that the even numbered releases (D4 and D6) were buggy and their
odd-numbered successors (D5 and D7) were much better.
That's correct for the Borland products, up to D7. Afterwards bad goals
had been chosen, e.g. a competitor for Microsoft .NET, which failed
miserably. Then the adaptation to the MS (WinAPI) help files took
several years, and online help still is at <50% compared to D4 or D7.
Next came the introduction of Unicode support and Firemonkey, and way
too late a 64 bit compiler.
Post by P E Schoen
I found some
Delphi Enterprise 8 products on eBay for under $200 and CodeGear Delphi
2007 Enterprise for $430. Embarcadero does not offer an upgrade deal
from D4, and their lowest level (Pro) version that offers database
support is $1405.
https://www.embarcadero.com/app-development-tools-store/delphi
Upgrading existing projects seems to be quite easy, at least I could run
a D5 program very soon on XE and XE4, with only a few missed detail
adaptations found later. The prices, well, I think they are due to a
shrinking user base. New projects mostly use VS, but that's not an
option for your existing Delphi projects.
Post by P E Schoen
Perhaps even better is the open source Free Pascal and the Lazarus IDE
that has a look and feel very similar to the Borland Delphi IDE that I
am used to with D4-Pro.
http://wiki.freepascal.org/Lazarus_Documentation
https://www.youtube.com/playlist?list=PLB24C56953A79987A
I discontinued watching FPC and Lazarus development. Some Delphi
projects may port easily, others can require much work.

DoDi
P E Schoen
2016-08-22 11:12:46 UTC
Permalink
Post by P E Schoen
Perhaps even better is the open source Free Pascal and the Lazarus IDE
that has a look and feel very similar to the Borland Delphi IDE that I
am used to with D4-Pro.
http://wiki.freepascal.org/Lazarus_Documentation
https://www.youtube.com/playlist?list=PLB24C56953A79987A
I discontinued watching FPC and Lazarus development. Some Delphi projects
may port easily, others can require much work.
I installed them on my Win10 computer and I have had some success with
simpler projects. I'm having trouble installing the ComDrv32 package, but I
don't fully understand how they work in Delphi, although I was able to do it
somehow. I was able to get around the problem by removing the component from
the form and then manually creating the component.

Now that I think about it, I probably have two instances of the component in
my project, which might cause problems.

Thanks,

Paul

Hans-Peter Diettrich
2016-08-19 10:27:25 UTC
Permalink
Post by P E Schoen
I have an application that uses a serial port component to read and
write data through a USB serial port at 57.6 kb. Originally I used the
SerialNG component and with some tweaking I was able to get it to work
reliably. But it has problems with Win10, so I changed the code to use
ComDrv32, which works when used in a simple TTY demo.
I don't remember what prevented programs from using the asynchronous I/O
API. 57kbd also doesn't look like a dangerous transmission speed.
Post by P E Schoen
I had to use a separate PChar pointer RecdPtr, and I also had to use an
integer for the pointer arithmetic. Apparently Delphi does not allow
RecdPtr := RecdPtr + (DataSize - (MAXCOMM-CommBufferPtr) ); RecdPtr + n
is OK. I'm not sure if I could just advance the DataPtr in the same way,
since it is declared as a generic Pointer type.
Delphi allows pointer math only with PCHAR. At least I'd use Cardinal
instead of Integer, in a workaround.

DoDi
Loading...