Serial Port Handling

[Astro]

Hi,

This is more of a general question than MASM specific, but figured here was as good a place to ask as any.

I'm trying to figure out a few things about how Windows handles serial ports. It seems there is an internal (to Windows) 4kB buffer, but I need to do a few things:

* Increase this buffer size
* Determine when it is full myself
* Control the serial lines directly

In addition to the in-built buffer, as part of ReadFile/WriteFile I have to point to an address of MY buffer that is used for sending/receiving data as appropriate.

One problem I have had, is just writing a simple asynchronous loop to read a serial port (and doing nothing else in the program) results in me losing data (I get odd output). Whilst I am unable to read data reliably, there is no hope in hell of me writing and knowing whether it worked as things are a fair bit more complex than simple serial data (and I've no easy way to read the writes though apparently there are com port sniffers that run on the system that can do this), however, the reads were performed on a straight-forward serial port and they were screwing up.

Please bear with me regarding source code - I need to dig out my dev work I did so far (it has been on hold for a while). There were some posts I was directed to a while ago on this - I shall go take a look.

Thanks in advance.

Best regards,
Robin.

[redskull]

It's unlikely you will be able to change the size of this buffer, and certainly not control serial port pins directly.  A better question is why 4K isn't big enough to begin with; the only purpose of that buffer is to hold the data for the duration between when the device completes and your thread is scheduled again.  You should be emptying that buffer every time through, and never even come close to filling it up.  But without more information of what you are doing or how you are attempting it, there's not too much that can be done.  Are you still doing this? http://www.masm32.com/board/index.php?topic=13693.msg107516#msg107516

-r

[Astro]

Hi,

Are you still doing this?

Yes I am.

As far as I can tell I need to permit each VCP to run at 115k2 each, at the same time as the real, single port running at 115k2. I think you can see the problem...

The VCPs are going to be backing up data like no tomorrow. I have no control over what is connected there, and right now the only solution I can see working is to run each VCP 1/4 the speed of the real port. I'm not sure if this degradation is acceptable. If not, we've got problems (need to find a new chip).

Best regards,
Robin.

[clive]

We'll I've run 460800 baud over USB (USB-to-Serial Dongle) with the generic Win32 serial api without altering the buffer size or doing anything clever. I pull the data in/out of ring buffers, and process it out of there. I use state-machines, but you could probably do it with threads, you just have to be ready to shovel data quicker than it can accumulate.

One thing to watch with the ReadFile/WriteFile interface is to look at the dwRead/dwWrite parameters very closely as the interface will not always give/take what you want to give it all the time. So take them into account, advance the buffer, and try again. Say you try to send 1KB at 9600 baud, the WriteFile will likely return success, but only take ~900-1000 bytes of it.

Use a protocol that lets you modulate flow or packetize the data (ie XMODEM, SLIP, HDLC) intrinsically.

You can control RTS/CTS and DTR/DSR for hardware flow control (ie Modems), but a lot of cables aren't wired right

[redskull]

As far as I can tell I need to permit each VCP to run at 115k2 each, at the same time as the real, single port running at 115k2.

Obviously, in the worst case, you can only run the real com port at 1/n the speed of the virtual ones, where n is the number of VCPs.  However, since the virtual ones are not really constrained by physical limitations of com port hardware (they are, after all, just copying data around in memory), you might be able to run them as fast as you need, or replace them using pipes or something.

[Astro]

Say you try to send 1KB at 9600 baud, the WriteFile will likely return success, but only take ~900-1000 bytes of it.

Ahhhhhhh......... this might explain the weirdness with my port reads? If so, then my only response to that is: WTF?

Obviously, in the worst case, you can only run the real com port at 1/n the speed of the virtual ones, where n is the number of VCPs.  However, since the virtual ones are not really constrained by physical limitations of com port hardware (they are, after all, just copying data around in memory), you might be able to run them as fast as you need, or replace them using pipes or something.

Pipes aren't an option - they must be COM ports as far as the system is concerned (unless I mis-understood you).

Best regards,
Robin.

[clive]

this might explain the weirdness with my port reads? If so, then my only response to that is: WTF?

For the reads this is pretty much a given, as it will timeout and I've always checked dwRead, but for writes I was being lazy assuming that it was blocking anyway and would take all I could give it. So yes my response was also WTF, and resulted in me patching quite a number of simple applications. I would hazard that it relates to baud rate, and the current buffer fill levels. I really hate to micro-manage the serial port at this level, the application really shouldn't have to care what the baud rate, data buffers, or the man behind the curtain are up to. Give me what you have, take what I'm giving you, and don't loose it, or screw it up.

To expand a little from Red's observations, when consolidating or tunnelling serial data, if the baud rates or data flows can be variable it's always good to implement some elastic data buffers (ring buffers of sufficient depth, or pipes like Red suggests) to mitigate the ebb and flow of data, bursts, and general disparities in baud rates, even small ones (baud clocks generate by different sources can be very sloppy).

I would say you *always* want some intermediate buffering scheme, copying directly from one serial port to another is not a good plan for success.

Here's what I've done :

Code Select Expand

BOOL Serial_SendBuffer(SERIAL *Serial, DWORD Size, BYTE *Buffer)
{
DWORD dwWrite;
  BOOL Status;

  while(Size)
  {
    dwWrite = 0;

    Status = WriteFile(
      Serial->hFile,  // handle to file to write to
      Buffer,         // pointer to data to write to file
      Size,           // number of bytes to write
      &dwWrite,       // pointer to number of bytes written
      NULL);          // addr. of structure needed for overlapped I/O

    // Depending on the baud rate the buffer may not accept the entire
    //  payload. For 9600 baud a limit of 880 chars has been observed.

#ifdef DEBUG
    printf("%d %4d %4d\n",Status,dwWrite,Size);
#endif

    if (!Status) // Failure is hard
      break;

    Buffer  += dwWrite;
    Size    -= dwWrite;
  }

  return(Status);
}


[Astro]

Hi,

Thanks for that. I tried both blocking and non-blocking calls to ReadFile (I ultimately wanted it to be non-blocking using overlapped I/O and WaitForSingleObject - this should enable other threads/calls to work the port?).

// Depending on the baud rate the buffer may not accept the entire
    //  payload. For 9600 baud a limit of 880 chars has been observed.

Useful!! I don't think they put THAT in MSDN...  :boohoo:  To help this debug process I'm finding out if I can make the port simply echo anything it receives. This will help find any issues such as this.

It is a bit more work than I was hoping for, but my master plan was to stick the reads into separate threads. If data arrives, then it does a read of the buffer and writes it out to the relevant port, then sits and waits again, so the writes should occur very soon after the data arrives.

I was going to have one thread per port, then they can hopefully do whatever they like, have them call the handler code as required (I need to analyze each packet as some requests may be port management requests etc. for the CMUX).

I would say you *always* want some intermediate buffering scheme, copying directly from one serial port to another is not a good plan for success.

I was going to implement a ring buffer as you suggest. My issue at the moment though is I can't read the port reliably. I have tried writing but I'm not sure that is working properly (and the bad reads don't help, though the stuff I do get back doesn't appear to change as a result of the write, so I can only assume the write failed).

I thought it was going to be easier than this!  :eek

Best regards,
Robin.

[Astro]

One thing to watch with the ReadFile/WriteFile interface is to look at the dwRead/dwWrite parameters very closely as the interface will not always give/take what you want to give it all the time. So take them into account, advance the buffer, and try again. Say you try to send 1KB at 9600 baud, the WriteFile will likely return success, but only take ~900-1000 bytes of it.

I re-read and I see what you mean (didn't get it the first time). I'll check I'm doing this.

Best regards,
Robin.

[clive]

// Depending on the baud rate the buffer may not accept the entire
    //  payload. For 9600 baud a limit of 880 chars has been observed.

Useful!! I don't think they put THAT in MSDN...

Worse it's probably platform dependent too. This was observed under XP Pro 32-bit while trying to debug an X/Y-Modem implementation, higher baud rates had worked fine. I ended up recoding this routine, which was initially much simpler, commenting it, and finding every instance of this code in my collection and fixing it. Many of the instances had worked fine with the original code, basically because the usage patterns were different. Another suggestion is to use a single subroutine, or macro, to do ALL the reads/writes with common code, abstracting it might be marginally less efficient, but beats having to fix the same problem in a dozen places and missing some.

The Win32 API, and Microsoft's handling of hardware, have plenty of these types of traps. As I told one of my colleagues, if we just had to write paint programs under Windows this would be a breeze. Things turn really ugly if you start adding hardware, drivers, platform, user equipment (processors, speed, motherboards), and users into the mix.

You can goose the timeouts and buffer settings for the serial ports, people often use these first to mask underlying problems, hand that code to a million people, or even 10,000 and the real problem will keep biting you.