Tops-20 Kermit-20 Batch Test Battery Table of Contents Introduction 2 Triggered Sequential Testing 2 Speed Measurement Considerations 3 Loading Considerations 4 K20NUL: Basic Kermit-20 Server functional checkout 5 K20NUP: Basic Kermit-20 Server functional checking using PUT 5 K20PTY: Kermit-20 Server Transfer additional and post transfer checks 5 K20NRV: Kermit-20 Server functions via Tops-20 DECnet NRT to local host 5 K20NRT: Kermit-20 Server functions via Tops-20 DECnet NRT to remote host 6 K20NOL: Basic regression test against Kermit-20 4.2(174) [2-May-85] 6 K20POL: Regression test against Kermit-20 4.2(174) [2-May-85] using GET 6 K20PNW: Regression test against Kermit-20 4.2(174) [2-May-85] using PUT 6 K20NUR: Basic Kermit-20 Server functional checkout with parity 6 K20POR: Regression test against Kermit-20 4.2(174) [2-May-85] with parity 6 K20B8T: Kermit-20 Server 8-bit file transfer and post transfer checks 7 K20B8P: Kermit-20 Server 8-bit file transfer with parity and post transfer checks 7 K20B8A: Kermit-20 Server terminal 8-bit file transfer with parity and transfer checks 7 K20P8R: Regression test against Kermit-20 4.2(174) [2-May-85] 8 bit files with parity 7 K20RDC: Communications comparison Kermit-20 4.2(174) [2-May-85] and 5.3(243) 7 K20DPD: Kermit-20 Packet Decoding Example 7 K20TCP: Kermit-20 Transmit/Capture Testing via pseudo-terminal 7 K20TCN: Kermit-20 Transmit/Capture Testing via DECnet NRT 7 K2036P: Kermit-20 36 Bit Mode via pseudo-terminal 8 K2036C: Kermit-20 36 Bit Mode via pseudo-terminal with parity 8 K10NRT: Kermit-10 Regression Tests via DECnet NRT 8 K10NRP: Kermit-10 Parity Regression Tests via DECnet NRT 8 K1036N: Kermit-10 36 bit Regression Tests via DECnet NRT 9 Introduction ============ Before any general release is contemplated, Kermit-20 must pass a battery of 23 separate regression tests which exercise all available functionally that can be tested under via pseudo-terminal (also known as loopback testing) as well as DECnet NRT transport. In addition, Kermit-20 must communicate successfully with C-Kermit using TCP/IP based virtual terminals (also known as TVT's or Telnet) and properly handle a SPACE, PWD, and DIRECTORY command as well as a GET and PUT of both 7 text and 8 bit binary files. Remarkably, while many macros exist for C-Kermit, none of them appear to test such basic functionality and these tests are run manually in the course of normal packaging. Unfortunately, not all functionality can be tested. Currently, no actual physical serial hardware is available to test Kermit-20 from. It is believed that Kermit-20 will still function as a great deal of effort was expended to not break serial support by simulating a physical terminal as much as possible with pseudo-terminals. The protocol itself, of course, has no idea of what kind of connection it might be running on. The following batch tests and their resulting log files are provided as a courtesy and it is hoped that Kermit-20 enthusiasts will run them on their own systems. Timings should be taken with a gigantic grain of salt as a large number of factors can skew them. They are at best suggestive, and are only used for comparison with previous implementations on the same platform. Triggered Sequential Testing ---------------------------- The tests are entirely separate and there is no requirement to perform them in the order shown. That being said, three .MIC files are provided to queue related batteries for execution, the differences being whether regression testing is being done against older Kermit-20's and which kind of regression being run. The first, K20ALL.MIC, submits a battery of fourteen separate tests: K20NUL, K20NUP, K20PTY, K20NRV, K20NRT, K20NUR, K20B8T, K20B8P, K20B8A, K20DPD, K20TCP, K20TCN, K2036P, and K2036C for sequential execution. These test new and reimplemented functionality and all together, should not require more than two minutes and 8 seconds (00:02:00) of processor time. It is expected that the total wall duration should be less than a quarter of an hour wall time, a recent run taking fourteen minutes and forty eight seconds (00:14:48). The second, K20174.MIC, submits a battery of six separate tests: K20NOL, K20POL, K20PNW, K20POR, K20P8R, and K20RDC. These are regression tests against Kermit-20 4.2(174) from May 2nd 1985. Together, these should take no more than a minute of processor time (52 seconds) and an elapsed wall time of under a two minutes (00:01:21). The lesser total execution time of Kermit-20 4.2(174) of 22.7 seconds should be ignored as it is being asked to do less (such as not logging). The third, K10ALL.MIC, submits a battery of three separate tests: K10NRT, K10NRP and K1036C. These are regression tests against Kermit-10 V3(136) running on Tops-10 7.04 NET. Timings are 2:47 processor and 2:14:57 wall time. These figures are suspect due to known timing issues with TOPS-10 and the simh KS10 micro-engine. Speed Measurement Considerations -------------------------------- Differences in speed are seldom worthy of comment unless some situation yields truly abysmal throughput although this is typically due to simulator quirks. Otherwise, Kermit-20 is about as efficient and fast as it can be, over a years' worth of development effort having been expended in this area. For example, as of the time of writing (late 2023), Kermit-20 was tested sending a local file of 1,750,952 seven bit ASCII bytes to /dev/null on Mac OS X 13.5.1 using C-Kermit Beta 10 (numeric 20230924) with a packet size of 9,000 bytes (the maximum size possible). C-Kermit reported a data rate of 234.44 KC/s (or 2.289 MBd), the entire transfer taking 7.294 seconds. In contrast, Kermit-20 measured a total characters per second figure of 180.25 KC/s and an effective data rate of 2.1575 MBd, taking 7.93138 seconds to complete. Why are these numbers not the same? It's not a completely answered question, but one factor in the difference is probably that C-Kermit and Kermit-20 are considering differing points in time as the beginning and end of the transfer. Also, the reason Kermit-10 is reporting 2.1575 MBd as the baud rate is because this is the effective baud rate, taking expansion of compression sequences into account. Otherwise, the corresponding baud rate to 180.25 KC/s would be 1.76 MBd. The point is that making sure the measurements are analogous is not an immediately obvious or even intuitive decision. More important, C-Kermit is effectively seeing coaxial cable based Ethernet transfer rates of the early 1980's. If you really need speed, then the thing to do is consider implementing sliding windows. K20RDC does do such an apples-to-apples comparison, although its purpose is to validate character counts. That being said, the same set of 11 files totaling 85,840 characters was sent both the old and new versions of Kermit. With Kermit-20 4.2(174) doing the sending, a throughput of 66.6436 KC/s (580.274 KBd, effective) was achieved for a total duration of 1.44463 seconds. The newer Kermit-20 took 1.16638 seconds to yield a throughput of 82.5429 KC/s (718.7034 KBd effective), an increase of 23% (15.8993 KC/s). Of more interest, perhaps, are the processor time differences, Kermit-20 4.2(174) taking 600 milliseconds, the newer Kermit-20 taking 400 millisecond, a reduction of a 1/3. Again, this is anything but a fair comparison as Kermit-20 4.2(174) dates to May 2nd, 1985, a very different time. Processor utilization was simply not a factor as 2400 baud dial up was just being deployed, 1200 baud being the most common high speed dial up. However, even at the typical Columbia campus connection of 4800 baud, Kermit use did not present a significant load. Loading Considerations ---------------------- While it may be possible to run these jobs in parallel, it is very strongly and most emphatically recommended that you not do this as a load test. * First, and of critical importance, the temporary file names that are generated are not unique, making it possible for one test to overwrite the working results of another. * Second, because you will then be executing on a shared system, it will distort internal timing measurements, making comparisons with previous code execution speeds dubious at best. * Third, putting certain simulators under extreme load has triggered false hardware errors and could bring out other timing bugs, yielding false negative to the tests or even crashing the system. The User Environment Testing Pack (UETP) should be used for load testing and system checkout. K20NUL: Basic Kermit-20 Server functional checkout ================================================== 1) Pseudo-terminal login 2) INPUT statement test with C escape sequences 3) OUTPUT statement test with C escape sequences 4) Basic text file transfer checking 5) Wildcard file transfer checking to NUL: 6) Wildcard file transfer checking with different checksums 7) Session log testing 8) 94 character packets (largest for basic protocol) 9) Basic SERVER command repertoire 10) Basic NUL: tests K20NUP: Basic Kermit-20 Server functional checking using PUT ============================================================ 1) Like K20NUL, except uses push to remote NUL: instead of GET to local NUL: K20PTY: Kermit-20 Server Transfer additional and post transfer checks ===================================================================== 1) Prompt defaults (DECnet node if remote) and parsing 2) Time-out parsing 3) 120 character packets (larger then basic minimum, somewhat faster) 4) Internal transport timing checkout 5) INPUT/OUTPUT statement tests with C escape sequences 6) Pseudo-terminal login with no secondary fork 7) Basic macro definition check 8) Basic SERVER command repertoire 9) Basic NUL: tests 10) Short duration communications check: 1.62 milliseconds! 11) Basic text file transfers 12) Comparison of statistics between both sides (they're 'close'...) 13) Large text file test: Quantity Type -------- ---- 1,794,319 ASCII (7 bit) bytes 701 Tops-20 disk pages 9,000 Packet size (largest possible) 101.8023 KC/s (Highest total characters/second) 1.2186 MBd (Highest effective data rate) 14) Post transfer file comparisons (all OK) K20NRV: Kermit-20 Server functions via Tops-20 DECnet NRT to local host ======================================================================= 1) Same as K20PTY except 94 byte packet size only 2) DECnet NRT connection to the local Tops-20 host 3) Checks files, post transfer (all OK) 4) Does not include large file case 5) Fastest possible NRT (all communications are memory only) 6) Additional program information (memory layout) K20NRT: Kermit-20 Server functions via Tops-20 DECnet NRT to remote host ======================================================================== 1) Similar to K20NRV except going to a different Tops-20 host 2) Shows Flow-Control being cleared 3) Shows smaller packet sizes (which may be faster under certain simulators) 4) Slower transfer speeds (suspected to be an emulator artifact) 5) Example of BYE command 6) Example of /TIMEOUT K20NOL: Basic regression test against Kermit-20 4.2(174) [2-May-85] =================================================================== 1) Similar to K20NUL with reduced functionality because 4.2(174) does not have: a) No remote NUL: testing b) No PWD testing c) No remote statistics d) No large file/buffer e) No pseudo-efficiency simulations 2) 4.2(174) reports disk quota of 70,000 as "+Inf" 3) Compression appears to be somewhat better (could be incorrect, however) 4) Session logging test 5) N.B., 174 decimal is 256 octal K20POL: Regression test against Kermit-20 4.2(174) [2-May-85] using GET ======================================================================= 1) Similar to K20PTY with reduced functionality 2) Otherwise, similar to K20NOL, except for 3) 3) Post transfer file comparisons (all OK) K20PNW: Regression test against Kermit-20 4.2(174) [2-May-85] using PUT ======================================================================= 1) Same as K20POL, but uses PUT instead of GET K20NUR: Basic Kermit-20 Server functional checkout with parity ============================================================== 1) Like K20NUL (Basic Test), but using parity on packets 2) Somewhat slower transfer, perhaps 5% impact (may be false negative) K20POR: Regression test against Kermit-20 4.2(174) [2-May-85] with parity ========================================================================= 1) Like K20POL with parity checking 2) ASCII 7 bit files K20B8T: Kermit-20 Server 8-bit file transfer and post transfer checks ===================================================================== 1) Like K20PTY, except 8 bit files K20B8P: Kermit-20 Server 8-bit file transfer with parity and post transfer checks ================================================================================= 1) Like K20B8T, except does 8 bit files with even parity K20B8A: Kermit-20 Server terminal 8-bit file transfer with parity and transfer checks ===================================================================================== 1) Like K20B8E, except also checks parity from Tops-20 terminal driver K20P8R: Regression test against Kermit-20 4.2(174) [2-May-85] 8 bit files with parity ===================================================================================== 1) Like K20B8P and K20B8T, except does 8 bit files with even parity K20RDC: Communications comparison Kermit-20 4.2(174) [2-May-85] and 5.3(243) ============================================================================= 1) Regression test to send the same file and check character counts 2) Shows use of ECHO /DEBUG to put messages into log files K20DPD: Kermit-20 Packet Decoding Example ========================================= 1) Enhancement to decode a packet instead of dumping raw data 2) PTY transfer checks for fineness of time logging 3) Will use high precision Time of Day functionality(HPTIM% .HPTOD), if available 4) Otherwise, unmodified Tops-20 design limits time of day resolution to a little less than a 1/3 of a second (0.32958984375 seconds). In this case, Kermit-20 will attempt to construct millisecond resolution. 5) Shows timing resolution and number of bytes written to log file. K20TCP: Kermit-20 Transmit/Capture Testing via pseudo-terminal ============================================================== 1) Basic transmit, no parity processing, captured to NUL: 2) Transmit, space parity processing, captured to NUL: 3) Transmit, mark parity processing, captured to NUL: 4) Transmit, odd parity processing, captured to NUL: 5) Transmit, odd parity processing, forcing a parity error 6) Transmit, full even parity, including Tops-20 terminal line generated 7) Post transmit comparison checking of large and small files K20TCN: Kermit-20 Transmit/Capture Testing via DECnet NRT ========================================================= 1) Same as K20TCP, except tests are performed over a Tops-10/20 type DECnet Network Remote Terminal 2) Since DECnet NRT's do not generate parity, test 6 is performed without terminal line parity checking K2036P: Kermit-20 36 Bit Mode via pseudo-terminal ================================================= 1) Same as K20PTY, except no large files, all files being executables 2) Demonstrate correct and enhanced transaction logging 3) FILCOM/E demonstrates transfers correct to bit level 4) Double checked against EXEC directory, CHECKSUM BY-PAGES 5) Pre and post transfer file byte sizes and counts visually compared to be correct K2036C: Kermit-20 36 Bit Mode via pseudo-terminal with parity ============================================================= Note, be aware that while BATCON will properly strip parity for .LOG files, performing a DO of the same control (.CTL) file as if it were a .MIC file can result in strange output on the local terminal as Kermit-20's ECHO command will force parity. This is a design feature which is used for checkout and debugging. Further information may be gotten from Kermit-20's extensive and up to date built-in help by typing HELP ECHO and also HELP SET PARITY. 1) Same as K2036C, but with parity sending on terminal and packets and checking being performed 2) Demonstrate correct and enhanced transaction logging which is transparent to parity 3) Demonstrate proper error handling when a file cannot be opened by the Kermit-20 server; a. That server state is updated, b. That an 'E' packet is sent, c. That the 'E' packet is properly interpreted by the Kermit-20 client, and d. That no-remnants of the file are left in the testing directory. K10NRT: Kermit-10 Regression Tests via DECnet NRT ================================================= 1) Remote terminal tests (LOGIN, SYSTAT, Etc.) 2) Basic server tests, REMOTE SPACE, REMOTE HELP 3) REMOTE DIRECTORY, REMOTE DELETE validation 4) Kermit-10 does not implement CWD/PWD (as Tops-10 does not have the idea of a connected directory) 5) Transfer and comparison of various files, some quite large (all OK) 6) Session logging K10NRP: Kermit-10 Parity Regression Tests via DECnet NRT ======================================================== 1) As K10NRT, except REMOTE DELETE not validated and remote terminal assumed to work 2) Transfer and comparison of various files, none particularly large 3) Errors a. Correct parity not sent in response to FINISH b. Correct parity not sent in wildcarded directory listing K1036N: Kermit-10 36 bit Regression Tests via DECnet NRT ======================================================== 1) Test 36 bit mode against only other 36 bit client, a. Kermit-10 version 3(136) b. Tops-10 7.04 NET (KS10) 2) Comparison check against three executables on SYS: (DSKA:[1,4]) Name Extension Blocks Protection Date Version MACRO EXE 108 <055> 1-Sep-88 53B(1247) CREF EXE 28 <055> 1-Sep-88 53C(101) FILCOM EXE 28 <055> 1-Sep-88 22A(121) 3) Example of local session log 4) Example of local transaction log