26 Nov
2022
26 Nov
'22
7:47 p.m.
On Nov 26, 2022, at 12:40 PM, Thomas DeBellis
<tommytimesharing(a)gmail.com> wrote:
This is an interesting thread. Let me see if I can untangle it enough to glean the
understanding I'm looking for. I believe there is some amount of vagueness on my part
which is due to not understanding the correct question to ask nor how to properly phrase
the thing. I think another conceptual problem is that I tend to think API instead of
protocol, which can confuse the matter.
Again, the vast bulk of my programming experience has been with TCP/IP and those related
RFC's, some of which we ported to DECnet (SMTP comes to mind).
The question here that I'm converging on is, "when do you return from the API
call"? In TCP/IP, I'm not sure it is as well defined as I would like.
Nor is it in DECnet. As I pointed out, the architecture specs are quite deliberately and
quite clearly ONLY protocol specs. They specify what conforming implementations must send
or are allowed to send, and how conforming implementations are supposed to handle arriving
packets.
What the application interface looks like (or whether there is one in the first place) is
entirely unspecified. And in fact the implementations span a wide range.
For example, as Johnny reports, RSX considers a transmit complete when the packet is
acknowledged by the other end, which implies it's available to the destination
application. RSTS considers it complete when the packet has been queued for transmission
locally (i.e., the outbound queue limit has not been exceeded and an outbound buffer was
available for the data).
Most implementations handle flow control internally, but RSTS exposes it in the API. Most
implementations offer a "session layer message" API, but RSTS has a segment API
-- so in RSTS, segmenting a message into segments small enough to fit the negotiated NSP
segment size is up to the application, not the DECnet syste code below the API, and ditto
the reassembly at the far end. And while one might certainly quibble about the wisdom of
such design choices, they are perfectly legal according to the specs.
A comparison with FTP is somewhat illustrative,
although once again, DAP and FTP are vastly different things. For FTP, the control port
is actually running RFC 854 TELNET, so you know you are going to wake up on every line
break. Once negotiated, the data port is a stream of bytes which may or may not have
blocking in it. You wake up early from a read in that case because the sender closes the
port at end of file.
Paged file structures completely broke this by holding the data port open, so you have no
implied end of file, but have to poll for data, which is disgusting. That's why I
know about PSH because I use it to try to guess end of stream. It's a hack.
That being said, I think where I got confused was understanding the difference between
the various levels of buffering, blocking and enforcing record structure. It's
straightforward enough, but somewhat hairy in places and in others, perhaps unfortunate.
In a transfer, data is buffered up in a linked list (the send queue) and flushed based on
a number of criteria. One example, is the number of files done. For large directories,
this can cause the memory usage to vastly expand as everything is sent at once. That had
caused memory issues, which edit 111 partially fixed (but is kind of a hack, as I
recall).
When emptying the send queue, Tops-20 DAP only sets end of record on the last message in
the queue, which was confusing. The receiving DAP appears to have to know that it may get
a number of records in one read. I think what's happening is that Tops-20 DAP is
signaling, "I'm done sending you some stuff, I won't send you any more and
I'm expecting you to respond"
Again, what's happening here is confusing to me, which indicates I ought review the
DAP specification again. I was going to do that, anyway to break out the flag names
(instead of just typing the bits)
I'm not sure what "end of record" means, that isn't in the DAP 5.6.0
spec.
If you try to make sense of DECnet application protocols from the viewpoint of TCP based
ones that sit directly on top of the TCP stream (i.e., they don't do a packet layer,
as many like iSCSI do) you're going to have a difficult time. In DECnet, NSP and the
session layer offer a packet service, and pretty much all the protocols are explicitly
built on top of packets, not streams. So in the DAP protocol you see messages being
exchanged, and the meaning of data is described in terms of the fields of a packet.
Applications may have their own blocks of data to worry about. For example, in DAP there
are two obvious ones: (a) files and (b) records. DECnet doesn't limit the size of a
message, but it would still be a bit strange for a file to come across as a single message
so DAP does not do that. But it does (at least typically, I'm not sure I know all the
special cases) package records one to an NSP message, when in record transfer mode. (In
block transfer mode, one block per message.)
If I were to port iSCSI or NSP to DECnet, those protocols would get simpler: both would
lose their framing sublayer and instead just carry application layer packets one per NSP
message.
paul