On 2015-01-08 21:57, Mark Pizzolato - Info Comm wrote:
On Thursday, January 08, 2015 at 12:22 PM, Johnny Billquist wrote:
Even better. This I can work with, and I can do the same...
Actually, I'll be tricky and start with RSX, and if that fails, try VMS, on the
client side. While the server should understand both.
Now I just need to figure out the binary blob. Could you capture one from a
plain text file transfer for me? I think that should be enough.
(Actually, any file, as long as you can also provide me with what a DIR /FULL
shows for the file.)
I have a very strong suspicion what is in there, I just want to verify it.
When I looked earlier there were a lot of zeros. Guessing is probably not worth it. I think it would be more productive to dig through the source to the HGFTP server. The source is in the HGFTP033.F saveset within the previously mentioned zip file. The module FTP_FTON.B32 seems to contain the relevant magic:
Thanks.
If you don't have the system to unpack the zip and extract these files from the saveset I'll be glad to send them directly. Otherwise I'm done digging.
This should be enough. The thing is, I was expecting it to be pretty much this. It's just a couple of magic values that needed to be identified.
You have done more than enough. I do not expect that you have to do any more. I eventually would just like to test/verify this against a live system. This weekend probably...
Johnny
On Thursday, January 08, 2015 at 12:22 PM, Johnny Billquist wrote:
Even better. This I can work with, and I can do the same...
Actually, I'll be tricky and start with RSX, and if that fails, try VMS, on the
client side. While the server should understand both.
Now I just need to figure out the binary blob. Could you capture one from a
plain text file transfer for me? I think that should be enough.
(Actually, any file, as long as you can also provide me with what a DIR /FULL
shows for the file.)
I have a very strong suspicion what is in there, I just want to verify it.
When I looked earlier there were a lot of zeros. Guessing is probably not worth it. I think it would be more productive to dig through the source to the HGFTP server. The source is in the HGFTP033.F saveset within the previously mentioned zip file. The module FTP_FTON.B32 seems to contain the relevant magic:
fileattr_buffer[FATTR_L_VERSION] = FATTR_C_FILEATTR_VERSION;
fileattr_buffer[FATTR_L_LENGTH] = FATTR_S_FATTRDEF;
fileattr_buffer[FATTR_L_FAB_L_ALQ] = .in_fab[FAB$L_ALQ];
fileattr_buffer[FATTR_L_FAB_L_FOP] = .in_fab[FAB$L_FOP];
fileattr_buffer[FATTR_L_FAB_L_MRN] = .in_fab[FAB$L_MRN];
fileattr_buffer[FATTR_W_FAB_W_DEQ] = .in_fab[FAB$W_DEQ];
fileattr_buffer[FATTR_W_FAB_W_MRS] = .in_fab[FAB$W_MRS];
fileattr_buffer[FATTR_B_FAB_B_ORG] = .in_fab[FAB$B_ORG];
fileattr_buffer[FATTR_B_FAB_B_RAT] = .in_fab[FAB$B_RAT];
fileattr_buffer[FATTR_B_FAB_B_RFM] = .in_fab[FAB$B_RFM];
fileattr_buffer[FATTR_B_FAB_B_BKS] = .in_fab[FAB$B_BKS];
fileattr_buffer[FATTR_B_FAB_B_FSZ] = .in_fab[FAB$B_FSZ];
fileattr_buffer[FATTR_B_XAB_B_RFO] = .in_xabfhc[XAB$B_RFO];
fileattr_buffer[FATTR_W_XAB_W_LRL] = .in_xabfhc[XAB$W_LRL];
fileattr_buffer[FATTR_B_XAB_B_BKZ] = .in_xabfhc[XAB$B_BKZ];
fileattr_buffer[FATTR_B_XAB_B_HSZ] = .in_xabfhc[XAB$B_HSZ];
fileattr_buffer[FATTR_W_XAB_W_MRZ] = .in_xabfhc[XAB$W_MRZ];
fileattr_buffer[FATTR_W_XAB_W_DXQ] = .in_xabfhc[XAB$W_DXQ];
fileattr_buffer[FATTR_W_XAB_W_GBC] = .in_xabfhc[XAB$W_GBC];
fileattr_buffer[FATTR_B_XAB_B_ATR] = .in_xabfhc[XAB$B_ATR];
fileattr_buffer[FATTR_B_FAB_B_RTV] = .in_fab[FAB$B_RTV];
fileattr_buffer[FATTR_W_FAB_W_BLS] = .in_fab[FAB$W_BLS];
Where in FTP.R32 the FATTR structure seems to be defined:
_DEF(FATTR)
FATTR_L_VERSION = _LONG,
FATTR_L_LENGTH = _LONG,
FATTR_L_FAB_L_ALQ = _LONG,
FATTR_L_FAB_L_FOP = _LONG,
FATTR_L_FAB_L_MRN = _LONG,
FATTR_W_FAB_W_DEQ = _WORD,
FATTR_W_FAB_W_MRS = _WORD,
FATTR_B_FAB_B_ORG = _BYTE,
FATTR_B_FAB_B_RAT = _BYTE,
FATTR_B_FAB_B_RFM = _BYTE,
FATTR_B_FAB_B_BKS = _BYTE,
FATTR_B_FAB_B_FSZ = _BYTE,
FATTR_B_XAB_B_RFO = _BYTE,
FATTR_W_XAB_W_LRL = _WORD,
FATTR_B_XAB_B_BKZ = _BYTE,
FATTR_B_XAB_B_HSZ = _BYTE,
FATTR_W_XAB_W_MRZ = _WORD,
FATTR_W_XAB_W_DXQ = _WORD,
FATTR_W_XAB_W_GBC = _WORD,
FATTR_B_XAB_B_ATR = _BYTE,
FATTR_B_FAB_B_RTV = _BYTE,
FATTR_W_FAB_W_BLS = _WORD,
FATTR_W_XAB_SEMANTICS_LENGTH= _WORD, ! sizeof(xab_stored_semantics)
FATTR_W_XFILL = _WORD, ! Preserve QUAD alignment
FATTR_X_XAB_STORED_SEMANTICS= _BYTES(XAB$C_SEMANTICS_MAX_LEN)
_ENDDEF(FATTR);
This implies that for some files there are extended attributes appended.
If you don't have the system to unpack the zip and extract these files from the saveset I'll be glad to send them directly. Otherwise I'm done digging.
- Mark
On 2015-01-08 21:06, Mark Pizzolato - Info Comm wrote:
On Thursday, January 08, 2015 at 10:44 AM. Johnny Billquist wrote:
Aha! That is really good information. Now I just wonder if it always tries this
trick, or only if the other side identifies itself as a VMS system. Could you do
the same test against a Unix system and see if it tries the STRU command
there too?
The binary blob is what I need to figure out next...
As of now, Text transfers to and from RSX works fine, and I've extended the
directory output a little. I'm wondering if I should bother trying to fool things
like web browsers next, as they also tries to be clever and tries to
understand a Unix-style directory output.
What I see is that the VMS FTP client sends the "STRU O VMS" command immediately after the connection is established (BEFORE the login is established).
Here is a conversation with a VMS (MultiNet) FTP client talking to a VMS (Multinet) FTP Server:
[...]
Even better. This I can work with, and I can do the same...
Actually, I'll be tricky and start with RSX, and if that fails, try VMS, on the client side. While the server should understand both.
Now I just need to figure out the binary blob. Could you capture one from a plain text file transfer for me? I think that should be enough.
(Actually, any file, as long as you can also provide me with what a DIR /FULL shows for the file.)
I have a very strong suspicion what is in there, I just want to verify it.
Johnny
On Thursday, January 08, 2015 at 10:44 AM. Johnny Billquist wrote:
On 2015-01-08 17:57, Mark Pizzolato - Info Comm wrote:
I did the Wireshark trace and doing a GET there is an initial exchange of
STRU O VMS and if the comes back with a "200 OK" the client knows that the
server is VMS and probably viceversa. Then the data delivered in the RETR
(separate TCP session as normal) is prefixed with a binary blob of data. The
format of that blob is important. I recalled that MadGoat FTP also
implemented this functionality (and interoperated with it - Matt Madison (of
MadGoat) ended up working for the MultiNet folks for a while). In any case,
over the years MGFTP became completely open source and is still
maintained by Hunter Goatley (the Goat part of MadGoat). The source is
available, but digging through the bliss code seemed a little harder than
asking Hunter Goatley directly which I've done. I'll let you know what he
says.
Aha! That is really good information. Now I just wonder if it always tries this
trick, or only if the other side identifies itself as a VMS system. Could you do
the same test against a Unix system and see if it tries the STRU command
there too?
The binary blob is what I need to figure out next...
As of now, Text transfers to and from RSX works fine, and I've extended the
directory output a little. I'm wondering if I should bother trying to fool things
like web browsers next, as they also tries to be clever and tries to
understand a Unix-style directory output.
What I see is that the VMS FTP client sends the "STRU O VMS" command immediately after the connection is established (BEFORE the login is established).
Here is a conversation with a VMS (MultiNet) FTP client talking to a VMS (Multinet) FTP Server:
<<< 220 vs4000.infocomm.com MultiNet FTP Server Process V5.4(17) at Thu 8-Jan-2015 12:26PM-PST
STRU O VMS
<<< 200 Stru O VMS ok.
USER mark
<<< 331 User name (mark) ok. Password, please.
PASS password
<<< 230 User MARK logged into $DISK3:[MARK] at Thu 8-Jan-2015 12:26PM-PST, job 20204ea7.
STRU F
<<< 200 Stru F ok.
PASV
<<< 227 Entering passive mode; use PORT (192,168,60,3,18,103)
LIST
<<< 150 List started.
<<< 226 Transfer completed.
STRU O VMS
<<< 200 Stru O VMS ok.
PASV
<<< 227 Entering passive mode; use PORT (192,168,60,3,18,137)
RETR xremote.patch
<<< 150 VMS retrieve of $DISK3:[MARK]XREMOTE.PATCH;1 (3550 bytes) started.
<<< 226 Transfer completed. 2642 (8) bytes transferred.
PASV
<<< 227 Entering passive mode; use PORT (192,168,60,3,18,207)
STOR xremote.patch
<<< 150 VMS Store of $DISK3:[MARK]XREMOTE.PATCH;2 started.
<<< 226 Transfer completed. 2642 (8) bytes transferred.
QUIT
<<< 221 QUIT command received. Goodbye.
The transfers in both directions had a 48 byte binary blob preceding the file data.
A conversation between the same MultiNet FTP client and a Unix host looks like:
<<< 220 ftp.NetBSD.org FTP server (NetBSD-ftpd 20100320) ready.
STRU O VMS
<<< 530 Please login with USER and PASS.
USER anonymous
<<< 331 Guest login ok, type your name as password.
PASS mickey at mouse.com
<<< 230-
<<< The NetBSD Project FTP Server located in San Jose, CA, USA
<<< 1 Gbps connectivity
<<< WELCOME! /( )`
<<< \ \___ / |
<<< +--- Currently Supported Platforms ----+ /- _ `-/ '
<<< | acorn[26,32], algor, alpha, amd64, | (/\/ \ \ /\
<<< | amiga[,ppc], arc, atari, bebox, | / / | ` \
<<< | cats, cesfic, cobalt, dreamcast, | O O ) / |
<<< | evb[arm,mips,ppc,sh3], hp[300,700], | `-^--'`< '
<<< | hpc[arm,mips,sh], i386, | (_.) _ ) /
<<< | ibmnws, iyonix, luna68k, | .___/` /
<<< | mac[m68k,ppc], mipsco, mmeye, | `-----' /
<<< | mvme[m68k,ppc], netwinders, | <----. __ / __ \
<<< | news[m68k,mips], next68k, ofppc, | <----|====O)))==) \) /====
<<< | playstation2, pmax, prep, sandpoint, | <----' `--' `.__,' \
<<< | sbmips, sgimips, shark, sparc[,64], | | |
<<< | sun[2,3], vax, x68k, xen | \ /
<<< +--------------------------------------+ ______( (_ / \_____
<<< See our website at http://www.NetBSD.org/ ,' ,-----' | \
<<< We log all FTP transfers and commands. `--{__________) (FL) \/
<<< 230-
<<< EXPORT NOTICE
<<<
<<< Please note that portions of this FTP site contain cryptographic
<<< software controlled under the Export Administration Regulations (EAR).
<<<
<<< None of this software may be downloaded or otherwise exported or
<<< re-exported into (or to a national or resident of) Cuba, Iran, Sudan,
<<< North Korea, Syria or any other country to which the U.S. has
<<< embargoed goods.
<<<
<<< By downloading or using said software, you are agreeing to the
<<< foregoing and you are representing and warranting that you are not
<<< located in, under the control of, or a national or resident of any
<<< such country or on any such list.
<<< 230 Guest login ok, access restrictions apply.
PASV
<<< 227 Entering Passive Mode (199,233,217,249,251,196)
LIST
<<< 150 Opening ASCII mode data connection for '/bin/ls'.
<<< 226 Transfer complete.
PASV
<<< 227 Entering Passive Mode (199,233,217,249,251,197)
RETR robots.txt
<<< 150 Opening ASCII mode data connection for 'robots.txt' (77 bytes).
<<< 226 Transfer complete.
QUIT
<<< 221-
<<< Data traffic for this session was 82 bytes in 1 file.
<<< Total traffic for this session was 3171 bytes in 2 transfers.
<<< 221 Thank you for using the FTP service on ftp.NetBSD.org.
- Mark
On 2015-01-08 17:57, Mark Pizzolato - Info Comm wrote:
On Thursday, January 08, 2015 at 8:38 AM, Johnny Billquist wrote:
On 2015-01-08 17:02, Paul_Koning at Dell.com wrote:
On Jan 7, 2015, at 10:43 PM, Johnny Billquist <bqt at softjar.se> wrote:
So the question is - what do VMS do when talking to another VMS
system, and how does VMS decide when to use the extra abilities?
If there isn t a spec, the alternative would be to capture an FTP session
with a tool like Wireshark, and reverse engineer things. That tends to be a
pain but it can be done if all else fails.
Yeah. At this point I suspect that would be the easiest. Sending a really small
file over should tell me enough, I think.
Anyone who have both a VMS client and server, and some machine
inbetween that can run tcpdump or similar, and who could get me a dump of
a simple file?
Please keep in mind that any passwords you enter in the session will also be
in such a log...
I did the Wireshark trace and doing a GET there is an initial exchange of STRU O VMS and if the comes back with a "200 OK" the client knows that the server is VMS and probably viceversa. Then the data delivered in the RETR (separate TCP session as normal) is prefixed with a binary blob of data. The format of that blob is important. I recalled that MadGoat FTP also implemented this functionality (and interoperated with it - Matt Madison (of MadGoat) ended up working for the MultiNet folks for a while). In any case, over the years MGFTP became completely open source and is still maintained by Hunter Goatley (the Goat part of MadGoat). The source is available, but digging through the bliss code seemed a little harder than asking Hunter Goatley directly which I've done. I'll let you know what he says.
Aha! That is really good information. Now I just wonder if it always tries this trick, or only if the other side identifies itself as a VMS system. Could you do the same test against a Unix system and see if it tries the STRU command there too?
The binary blob is what I need to figure out next...
Meanwhile, the HGFTP stuff is available at: http://vms.process.com/ftp/vms-freeware/fileserv/hgftp.zip
Thanks. Might be worth looking into.
As of now, Text transfers to and from RSX works fine, and I've extended the directory output a little. I'm wondering if I should bother trying to fool things like web browsers next, as they also tries to be clever and tries to understand a Unix-style directory output.
Johnny
On Thursday, January 08, 2015 at 8:38 AM, Johnny Billquist wrote:
On 2015-01-08 17:02, Paul_Koning at Dell.com wrote:
On Jan 7, 2015, at 10:43 PM, Johnny Billquist <bqt at softjar.se> wrote:
...Note where I said "page structure" above. :-) I've read RFC 959
(as well as the various additions) backwards and forwards way too many
times by now.
The page structure would be a way of doing this. It will definitely not work
with any other client out there. But on the other hand, I'm mostly interested
in the efficient transport of files between two RSX systems for this part
anyway, so it will be towards my own client only.
But I doubt VMS does it this way, and I was curious about if I could
possibly do something that would be compatible with VMS, since enough
commonality in file storage exists between RSX and VMS for this to actually
be useful.
So the question is - what do VMS do when talking to another VMS
system, and how does VMS decide when to use the extra abilities?
If there isn t a spec, the alternative would be to capture an FTP session
with a tool like Wireshark, and reverse engineer things. That tends to be a
pain but it can be done if all else fails.
Yeah. At this point I suspect that would be the easiest. Sending a really small
file over should tell me enough, I think.
Anyone who have both a VMS client and server, and some machine
inbetween that can run tcpdump or similar, and who could get me a dump of
a simple file?
Please keep in mind that any passwords you enter in the session will also be
in such a log...
I did the Wireshark trace and doing a GET there is an initial exchange of STRU O VMS and if the comes back with a "200 OK" the client knows that the server is VMS and probably viceversa. Then the data delivered in the RETR (separate TCP session as normal) is prefixed with a binary blob of data. The format of that blob is important. I recalled that MadGoat FTP also implemented this functionality (and interoperated with it - Matt Madison (of MadGoat) ended up working for the MultiNet folks for a while). In any case, over the years MGFTP became completely open source and is still maintained by Hunter Goatley (the Goat part of MadGoat). The source is available, but digging through the bliss code seemed a little harder than asking Hunter Goatley directly which I've done. I'll let you know what he says.
Meanwhile, the HGFTP stuff is available at: http://vms.process.com/ftp/vms-freeware/fileserv/hgftp.zip
In the days before we all had internet access, I had uucp transfers between VMS machines (via DECUS UUCP) which also preserved file attributes. I don't recall how I determined if the peer system was a VMS one, but I do recall that I also prefixed the file data with the attributes. The attributes I sent were a FDL (File Definition Language) text blob which I was then able to use directly with some FDL library API to create the desired file and I then did block I/O to populate the contents.
- Mark
On 2015-01-08 17:02, Paul_Koning at Dell.com wrote:
On Jan 7, 2015, at 10:43 PM, Johnny Billquist <bqt at softjar.se> wrote:
...Note where I said "page structure" above. :-)
I've read RFC 959 (as well as the various additions) backwards and forwards way too many times by now.
The page structure would be a way of doing this. It will definitely not work with any other client out there. But on the other hand, I'm mostly interested in the efficient transport of files between two RSX systems for this part anyway, so it will be towards my own client only.
But I doubt VMS does it this way, and I was curious about if I could possibly do something that would be compatible with VMS, since enough commonality in file storage exists between RSX and VMS for this to actually be useful.
So the question is - what do VMS do when talking to another VMS system, and how does VMS decide when to use the extra abilities?
If there isn t a spec, the alternative would be to capture an FTP session with a tool like Wireshark, and reverse engineer things. That tends to be a pain but it can be done if all else fails.
Yeah. At this point I suspect that would be the easiest. Sending a really small file over should tell me enough, I think.
Anyone who have both a VMS client and server, and some machine inbetween that can run tcpdump or similar, and who could get me a dump of a simple file?
Please keep in mind that any passwords you enter in the session will also be in such a log...
Johnny
On Jan 7, 2015, at 10:43 PM, Johnny Billquist <bqt at softjar.se> wrote:
...Note where I said "page structure" above. :-)
I've read RFC 959 (as well as the various additions) backwards and forwards way too many times by now.
The page structure would be a way of doing this. It will definitely not work with any other client out there. But on the other hand, I'm mostly interested in the efficient transport of files between two RSX systems for this part anyway, so it will be towards my own client only.
But I doubt VMS does it this way, and I was curious about if I could possibly do something that would be compatible with VMS, since enough commonality in file storage exists between RSX and VMS for this to actually be useful.
So the question is - what do VMS do when talking to another VMS system, and how does VMS decide when to use the extra abilities?
If there isn t a spec, the alternative would be to capture an FTP session with a tool like Wireshark, and reverse engineer things. That tends to be a pain but it can be done if all else fails.
paul
On Jan 8, 2015, at 6:10 AM, Sampsa Laine <sampsa at mac.com> wrote:
On 8 Jan 2015, at 13:07, Johnny Billquist <bqt at softjar.se> wrote:
...
Which text format, EBCDIC or ASCII? :P
The joke was more aimed at you stating that text files are the one universal format, which of course they are, until you meet IBM.
Or CDC which has 4 or more of its own, all different from either EBCDIC or ASCII.
paul
On 8 Jan 2015, at 13:59, Johnny Billquist <bqt at softjar.se> wrote:
Now, wait a minute.
So you have problems with GUI ftp clients and servers. So you go for kermit for file transfers. Which means going command line.
If you go command line, what's wrong with command line ftp, which will work without the silly issues of GUI tools?
You can create scripts for ftp as well...
And ftp deals just as well with the different types of files as kermit do.
Johnny
Well, basically, I don't really use FTP.
I mostly use SFTP when updating my websites, hosting setup etc, and for that I use a graphical client since the hosting is Unixy so everything works nicely - I just happened to try connect to a VMS box with Transmit.app and it was seriously borked, so I thought I'd mention it.
When I was mostly playing with Alphas and IA64 boxes, I'd use a set of scripts around command-line SFTP because it wasn't as broken as Transmit :)
Now I'm usually logged into the following:
- GORVAX over Telnet (VAX)
- HILANT over Telnet (VAX)
- CHIMPY over SSH (AXP)
Anyway, I don't run SSH on my VAXes (and something in somewhere* router is messing with the FTP, even in passive mode) so I started playing around with Kermit scripts and eventually found them so convenient that I now mostly use Kermit over SSH instead of the SFTP scripts.
Summa summarum: I use SFTP in normal situations for file transfer where available, most of the time. If it's not, I use Kermit over Telnet. And I run Kermit over SSH as well since it does no harm AND gives me the ability to pull or push files in-band should I feel the need to :)
Sampsa
* My current connection in Lebanon is NAT'd from the ISP, god knows what they might be doing to the FTP connection
