I'm about halfway done with the changes to the Tops-20 DAPLIB buffer scheme. Again, the point was to not run out of memory and crash when doing 'extraordinarily' wildcarded directory listings. Now instead of being able to buffer up 170 messages before sending, I can buffer up 10,375 messages, which is two decimal orders of magnitude better memory utilization (as he counts on his fingers). I haven't done any measurements, yet, however.

Part of the conceptual problem I had was that I had completely, utterly and thoroughly forgotten some basic terminology being used: 'record' and 'blocking', confusing the former with buffering and the latter with waiting to send.

No...

A record here is not a local Tops-20 memory buffer in the case of DAPLIB. In this case, it's actually BUFSIZ, which is the maximum size message that the remote system will accept, which is negotiated in the Configuration exchange.

The terminology for "blocking" here is more like what you do with magnetic tapes. Here, a record is the maximum size of data that the drive hardware will accept in a single I/O request. That is, you put 'blocks' of data into the tape record until its full. It has nothing to do with waiting. You might decide to hold on to that record and buffer up a few of them for better performance, using chained I/O. That's one way you keep a helical scan tape drive happy, for example.

What happens when DAPLIB dumps its internal buffers is that each block of data is written with a SOUT% until the last block is hit, which is written with a SOUTR%. SOUTR%'s job is to take all the buffered monitor data and push it over the wire. The remote reader wakes up either on a network record or input buffer full.

I can't remember how that works, how Tops-20 knows that DECnet transport has signaled the equivalent of a record break in the data flow.

Also, I noticed that Tops-20 starts out with a hardwired maximum BUFSIZ and was wondering whether it might be smarter to offer something else. In particular, once the link is in the RUN state, you can query the maximum segment size that can be sent with an .MORSS MOPR%. That value is defined to be the minimum of the maximum segment size supported by the remote NSP task, the segment size supported by the remote network task, and the segment size supported by the local Tops-20 NSP task. The local task can use this value to optimize the format of data being transmitted over the link.

Shouldn't I pay attention to that value instead of offering some big number?

On 4/25/23 9:59 AM, Thomas DeBellis wrote:

Thanks!! The RSX NFT /D switch reminds me that the list of features that I want to lift from RSX NFT and put into Tops-20 NFT just keeps getting longer...

Right now, I'm in the middle of changing the buffering mechanism for Tops-20 DAP to use correctly allocated sizes from a post-build static buffer header instead of a maximally allocated and trimmed dynamic buffer and then compacted post-build.

The latter current method only allows me to buffer up 170 messages, most of which are not very large at all. I would assume that I should be seeing far more than that given that I have 448 pages of buffer memory to play with. Admittedly, some of that is being taken up by linked list overhead, but it still seems like a flagrant waste of memory.

Of course, using that much main storage either way back in the day would almost certainly have gotten me 'a good talking to' (I.E., yelled at) for being a memory hog and paging the system into oblivion. One assumes that there would be a configurable parameter to use instead of what I'm doing (grabbing as much as I can get my hands on in the section zero address space).

On 4/24/23 9:51 PM, Johnny Billquist wrote:

A quick check on some different systems that are online on HECnet at the moment didn't reveal any with a bufsiz of 0 or greater than 4096.

I checked TOPS-20, Ultrix (a few different versions), VMS (including V9.2), RSTS/E and RSX.

With RSX, it's fairly easy to check:

.nft venti2::/id
NFT -- Version V9.0
Local NFARs V8   DAP V7.1 Buffer size= 2064. OS=RSX-11M+ FS=FCS-11 DC=Yes
Remote FAL V2   DAP V5.6 Buffer size= 4096. OS=TOPS-20   FS=TOPS-20 DC=Yes

Johnny

On 2023-04-25 00:13, Thomas DeBellis wrote:

After I got done 'enough' with the auto-mounting code for FAL (meaning, I didn't make the modifications to the access control job because I bumped into .MSIMF), I turned to fixing a bug that I bumped into a few years ago. While Tops-20 does not support wildcarding devices, it does support wildcarding all structures that the system currently has online via the DSK*: moniker.

So, let's say you're a 60+ who is having those occasional lapses of memory and you /really/ can't remember a single thing about something you think you might have somewhere. This is, of course, completely forgetting that you wanted to find something in the first place... you can ask the EXEC to do something like, DIRECT DSK*:<*>*SOMETHING*.*.0. You can also hand that to FIND to have it look in all those files.

Some experimentation with NFT shows that Tops-20 FAL supports structure wildcarding. BUT!! If you're being nosy or are just plain hacking and do DIRECT DSK*:<*>*.*.*, it crashes with a GLXLIB out of memory error (ASE, Out of Address Space)...

Researching this shows that Tops-20 DAPLIB tries to buffer messages before dumping them over the wire and that the decision it makes to dump is file count based. That appeared to not work properly because of two issues. First, each file was actually generating four separate messages: a Name, an Attributes, a Date/Time Attributes and a File Protection Attributes. Second, the buffer builder was always allocating a buffer for the maximum size message, 4090 characters (1032 thirty-six bit words).

That's pretty excessive, considering the typical sizes. Some instrumentation shows the below:

    Message     Allocated     Actual     %Excess    Name           4,090       18         99.56%     Attributes     4,090       17       99.58%     Date/Time      4,090       40         99.02%     Protection     4,090        7         99.83%

I changed the buffering algorithm to dump based on available memory. The starting available memory is 448 pages, so I take ¾ of that to devote to DAP buffers and ¼ (122 pages) to do the dumping. That fixed the problem. Then I changed the memory allocation to trim unused words from the end of messages. That's not working as well as I expected, probably because I need compact (or 'burp') the memory. Another alternative would be to build all messages in a single area and then only allow a chunk in the linked list for what I need. No burping.

A concern is what happens with the Configuration BUFSIZ operator and what to do if it is 0 (unlimited). The DAP specification specifies that an operand of zero means that if one of the two systems has an unlimited buffer size, it sends a 0 and the two systems will use the nonzero buffer size as the maximum. If both systems send 0, thereis no limit on the length of messages which may be sent.

My question is whether zero is absurd. Does any implementation send that?

It probably doesn't matter that much for sending, but I'm also thinking that I might hypothetically have a similar problem on receive. Obviously I don't because the current Tops-20 DAP won't negotiate a size larger than 4096 bytes. Is anything running larger than that? Would it be silly to?