Version 0.96
(This page best viewed with Firefox. Some characters are not rendered by IE6)
(c) Wally Bass 2009. Released to the Public Domain
This package contains the following DOS programs and XyWrite XPL program tools:
The XYENC raw encoding is an encoding that is devoid of both guillemets and of XyWrite “3 byte encoded data.” In its rawest form, the encoding is limited to only “pure ASCII” characters in the 20h to 7Fh range, plus CRLF pairs for separating lines. This makes raw encoded files readable, editable, transmittable, and generally manageable using tools that don't work with or have difficulties handling normal XyWrite files. In particular, the raw encoding allows for the transmittal of XyWrite files and XPL programs as text within the body of almost any email, using nearly any email handling facilities.
When used for the purpose of transmitting files, via email or other means, XYENC and XYDEC include provisions for end-to-end validity checking of the encode/transmit/receive/decode process, using the same CRC32 checking algorithm found in ZIP files.
When reading or editing a raw encoding, a user must be aware of a number of character translations done by XYENC. For example, in a raw encoding, a left guillemet is represented by the sequence “˜<” (tilde,less than).
Both raw and cooked encodings have the property that whitespace characters (blank, tab, CR, and LF characters) are not used in the encodings to encode input data. These characters are hence discarded during the decode process. The net result of this is that whitespace can be added to encoded files for much improved readability, without affecting the capability to decode the file back to the original. QDF1 is a XyWrite XPL program that adds whitespace to a raw encoding, using indentation to help show program structure for an encoded XPL program.
There are also provisions for adding comments to either raw or cooked encoded files, also without affecting the ability to decode back to the original. And, XyWrite embedded formatting command can also be added to encoded files without affecting the decode, allowing for even more elaborate formatting of translated files.
Although there are various degrees and flavors of “cooking,” we generally think of an encoding as no longer raw when the cooking has reintroduced XyWrite encodings that are only understood by XyWrite (such as 3 byte encoded data).
Relative to raw encodings, cooked encodings have the disadvantage of requiring XyWrite/NB (III+ or later) for viewing. Cooked encodings are also (probably) not suitable for email transmissions.
Relative to the ORIGINAL file, though, cooked encodings have the huge advantages of (a) allowing the addition of (or already having added) whitespace, to format an XPL program for readability (including indentation to reflect program structure), and (b) being usefully viewable in XyWrite normal (e.g., not “expanded”) display mode.
Like most “word processor” files, XyWrite files can be thought of as consisting of a mix of 'data' and 'metadata.' The 'data' consists of those characters in the file that we want the reader to see and read. The 'metadata' is other data that we usually want hidden from view, but is data that “advises” the “word processor” program (e.g., XyWrite) as to how the 'data' characters are to be displayed to the user. 'Metadata' is used to encode such information as margin settings, indenting information, and desired font information.
XyWrite (III+, for example) uses two strategies or mechanisms to deal with various metadata issues. The first mechanism is the use of the left and right guillemet characters, which are be used to “enclose” metadata sequences, much like parentheses are used to enclose parenthetical material in everyday writing. The second mechanism is the use of “3 byte encoded” representations of “characters.”
With the guillemet bracketing mechanism, XyWrite normally displays guillemet bracketed metadata as a small triangle occupying only a single character position onscreen. The triangle serves as a reminder that the metadata is present, but does so without unduly interfering with the desired display of the data.
Just as guillemet bracketing is used in normal XyWrite files to distinguish data from metadata, XyQuest chose to uses guillemet bracketing in XPL programs to distinguish literal data content (e.g., what would be “string constants” in most program languages) from XPL “programming language” and “program logic” content, such as IF statements and “assignment” statements which assign values to variables. While convenient in some cases and from some points of view -- in implementing simple “data macros,” for example -- this strategy fits poorly with the desire to sensibly format and display XPL programs for user review, analysis, and editing.
In most XPL programs, the “program logic content” comprises the majority of the XPL program. It is really the “program logic content” that IS the “data” that is of most interest to the user. When that is kept in mind, it is clear that thrusting the program logic content of XPL programs into the metadata handling facilities of XyWrite is something of a usability disaster. In normal (e.g., not “expanded”) display mode, XyWrite not only does nothing to format (or allow the user to format) the program logic content sensibly, but it hides the program structure content entirely, in its “metadata triangles.”
XPL programs are therefore only readable in XyWrite's expanded display mode, and in that mode, they get absolutely no onscreen formatting help from XyWrite. This deprives people working with XPL programs of what XyWrite normally does best -- to sensibly format data for readability, during both composition and subsequence rereading, analysis, and editing. This is one of the problems that the XYENC package seeks to address.
XyWrite's 3 byte encoding scheme is also largely addressed at metadata issues. This encoding scheme decrees that a byte with the value 0FFh in a file always marks the beginning of a 3 byte encoded sequence. XyWrite allows any “normal” data character to alternatively be represented with a 3 byte encoding -- a character whose 8 bit code value is 0MNh in hex (where M and N are hex digits) is “3 byte encoded” with a 0FFh byte followed by the two digits 'M' and 'N', encoded as printable ASCII characters (in uppercase, if the values are from “A” to “F”).
Some of the “normal” (1 byte) characters in the ASCII coding scheme are in fact not data, but are metadata characters whose metadata significance has been long established by traditional usage. An example is the 'tab' character, which doesn't actually display, but affects how subsequent data is displayed. The EOF (1Ah) character is another “normal character” which is really metadata. The Carriage Return (CR) and Line Feed (LF) characters are metadata characters with a long and often contradictory usage history in different OS environments.
In XyWrite, the 3 byte encodings of data characters do NOT represent metadata. On the contrary, what they do is provide an alternate way to represent ANY 8 bit character, but represent the character with an “understanding” that the encoded character is NOT to be treated as metadata, or to have any metadata “side effects.” Instead, such an encoded character is to be considered strictly and only as “real” or “visual” data. The decoding scheme used by XYDEC, and the “cooking” schemes that this decoding scheme enables, draws heavily on (and expands on) EXACTLY this concept, using XyWrite style 3 byte encoding to achieve a great deal of visual conformity between cooked encoded files and the original file content.
Encoding schemes for handling metadata involve “trigger” characters, which signal the beginning of encoded packets or “tokens.” As we have noted, XyWrite III+ has two trigger characters -- the 0FFh byte for 3 byte encodings, and the left guillemet character for XyWrite “embedded commands.” XyWrite IV adds the 0FEh byte as another trigger character. XyWrite III+ (at least) also uses 0FDh as a trigger character, but this particular usage is never reflected in the encoding of data that actually is stored in a file, except for the fact that XyWrite III+ (at least) will always use its 3 byte encoding of the 0FDh data character.
XYENC encoding adds several more trigger characters for encoding purposes: the “'” (right single quote, or rsquote), “˜” (tilde), “:” (colon), “;” (semicolon), and “!” (exclamation point, which we refer to simply as “exclaim”) are all trigger characters (to XYDEC, but not to XyWrite) in an XYENCoded file, in addition to the left guillemet and 0FFh bytes. The “_” (underscore) could also be thought of as a trigger character, but in this case, the “_” is the entire token or packet (it is translated to a blank).
Like XyWrite itself, XYDEC understands and makes use of XyWrite style 3 byte encodings to negate the special or metadata meaning of some characters. So, for example, while a “˜” (tilde) normally “triggers” XYDEC to look for an encoding like “˜ddd” (where the d's are digits, and the encoding is capable of specifying any character by specifying the byte value associated with its ASCII encoding) or “˜Z” (which represents Control-Z or 1Ah, the EOF character), a 3 byte XyWrite-encoded “˜” (tilde) is simply passed as a “˜” (tilde) character, and does NOT trigger an attempt to recognize and decode a tilde-prefixed token or packet.
XYDEC's handling of XyWrite style 3 byte encodings, coupled with the fact that XyWrite accepts these encodings and displays the desired character without complaint, is what gives rise to this package's ability to do a good deal of “cooking” of XYENCoded files, to improve readability.
(Strategies to separate data from metadata should be very familiar to anyone who has looked at HTML encoding for Web pages. HTML uses the “<” and “>” characters to identify and enclose metadata. This makes the “<” into a “trigger” character, giving rise the fact that an actual “data” “<” character must be represented in some other way in HTML. The “other way” chosen by HTML was to use the sequence “<” to represent the “<” character as data. This in turn make the “&” another “trigger” character in HTML, giving rise for the need to use the sequence “&” to represent an “&”. And so on.)
As was indicated earlier, XyWrite III+ uses the byte value 0FFh as a “trigger” character to recognize 3 byte encodings, and uses the left guillemet character to recognize “embedded commands.”
XyWrite IV adds to this the 0FEh trigger character, to recognize the encodings of 654 additional printable characters, in addition to the normal 256 printable characters represented by code page 437 byte values 0 to 0FFh. These encodings are of the form “0FEh,x,y” where x and y are bytes, and y:x forms a 16 bit word with values from 0000h to 028Dh.
XYENC/XYDEC adds to this list of “trigger” or similar “special” characters the “˜” (tilde), “'” (rsquote), “:” (colon), “;” (semicolon), “!” (exclaim), “_” (underscore), “%” (percent), “ ” (blank), “␉” (tab), “␍” (carriage return, or CR), “␊” (linefeed, or LF), and “␚” (01Ah, the EOF (End-of-File) character).
These characters have the following roles.
“%” (percent), “ ” (blank), “␉” (tab), “␍” (CR), “␊” (LF), and “␚” (EOF) characters are all ignored by XYDEC when present an encoded file. (They must, however, not be added in the middle of an encoded “token” triggered by the “'” (rsquote), “˜” (tilde), or “!” (exclaim) characters.) These character interpretations are what allow whitespace to be added to an encoded file, to format the file for improved readability, but without changing the decoding result for the file. The “%” (percent) character may seem a little peculiar in this role, but it's availability in a whitespace role allows it to be used as a “marker” in a variety of editing operations under XyWrite, without danger of the “%” being confused with normal data characters. The QDM1.PM XPL program takes advantage of this, for example.
“:” (colon) and “;” (semicolon) are used in an XYENC encoded file as a “trigger” or “flag,” to indicate data characters that were (in the original file) encoded with 0FFh triggered 3 byte encodings. “:” is used if the 3 byte encoding was of the XyWrite III+ variety (0FFh followed by ASCII encoded hex digits), and “;” is used if the 3 byte encoding was the new (with XyWrite IV) encoding used for special encoding of some search argument characters (0FFh followed by a byte 0C0h or greater, followed by another byte which is the encoded character XORed with 80h). To represent these 3 byte character encodings, XYENC merely uses a “:” or “;” in the encoded file, followed by the 1 byte version of the 3 byte encoded character. (The '1 byte version,' however might itself be subsequently expanded, according to XYENC's rules for encoding normal 1 byte characters.)
“_” is used by XYENC to encode blanks that were in the original file. Perhaps this should not be called a “trigger”, since the “_” byte is the entire encoding.
“!” (exclaim) is used to embed a supplemental packed of information in an encoding. These are required when a XyWrite 3 byte encoding was present in the original file, but the 3 byte encoding was not “properly” encoded in the “standard” way that XyWrite itself would use for the encoding. Although these 3 byte encodings are in some sense “errors,” XyWrite itself will accept many of these “erroneous” encodings without complaint, and these “erroneous” encodings are sometimes used intentionally in XPL programs to archive some hard-to-archive (with XPL) effects. XYDEC will recreate the “erroneous” encodings exactly as they were in the original file, but needs additional information (over and above the way that XYENC normally represents 3 byte encoded data) to do so. “!” (exclaim) triggered tokens are used to provide the additional information.
“'” (rsquote) and “˜” (tilde) are used as trigger characters for a variety of encodings. Many of these are encodings of the “trigger” characters themselves, which (when they appear as data characters in the original file) must be encoded in a XYENC encoding to keep them from being interpreted as triggers during decode. The following indicates the special encodings of this flavor:
Char Encode Char Encode Char Encode Char Encode
« ˜< blank _ ! '| _ '-
» ˜> CRLF '^ : '. ˜ '?
∈ ˜{ % '/ ; ', ' '`
≡ ˜=
To successfully read and understand XPL programs in XYENC raw encoded form, a user should become reasonably familiar with the above encodings.
In addition to the above, “'” and “˜” are used as triggers for the following general encoding “families”:
XYENC and XYDEC do character level encoding and decoding, taking into account the notion that XyWrite 3 byte encodings are regarded as single “characters.” Partially because of XyWrite's design, and partially because of XYENC and XYDEC's design, there are a number of different choices as to how some characters can be encoded. When there is a choice for how a character is encoded, any of the choices decode in the same way, so decoding is not affected by these choices. In some cases, the choice boils down to user preference.
For example, when considering the encoding of bytes with the values 01h to 06h, some might be familiar with the graphics ☺, ☻ ♥ ♦, ♣, and ♠ and hence might just prefer the native characters. Other might find it easier to relate XYENC's Unix-like ˜A, ˜B, ˜C, ˜D, ˜E, and ˜F encodings to the code values 01h to 06h. Similarly, considering the characters 80h through 85h, some might prefer the native characters Ç, ü, é, â, ä, and à, whereas others might prefer XYENC's encodings of ˜128, ˜129, ˜130, ˜131, ˜132, and ˜133. The merit of using the native character representations might also depend on the “code page(s)” that are available for use on the user's system, for displaying character codes greater than 80h.
XYENC and XYDEC really “don't care” which encoding you pick for (most) characters in the 0 to 1Fh and above 80h ranges. To meet the “all ASCII” specification, XYENC by default outputs the “˜A” type representations for the 0 to 1Fh range, but the 'L' invocation switch changes this to simply outputting the native characters for most characters in this range (but characters with metadata significance, such as tab, remain encoded). Similarly, XYENC by default outputs ˜128 type encodings for chars 80h and above, but will output native characters for most characters in this range if the 'H' invocation switch is specified.
For the output choices in these two ranges (0-1Fh, and 80-0FFh), I would think of the ˜A and ˜128 type encodings as the “raw” encodings, and the native character encodings as somewhat “cooked” encodings. For characters in these two ranges, the package provides no XPL programs to do “cooking” -- only the 'L' and 'H' options on XYENC are provided. But such XPL programs can be easily constructed (since they consist only a bunch of XyWrite “change” commands).
Usually of more interest and significance is the cooking of characters in the “normal” ASCII range of 20h through 7Fh, plus the commonly used (in XPL) left guillemet, right guillemet, “∈”, and “≡” characters. Here, the raw encodings include the “˜<”, “˜>”, “˜{”, “˜=”, “'/”, “'|”, “'.”, “',”, “'-”, “'?”, and “'`” encodings (see previous table) for «, », ∈, ≡, %, !, :, ;, _, ˜, and ' (respectively). “Cooking” of these encodings involves substituting 3 byte data encodings of the XyWrite variety, in lieu of the XYENC encoding. These then display as like the original character, under XyWrite. For most people, this will decidedly improve readability, when viewed under XyWrite. The QDM2.PM XPL program does this bit of cooking. QDM2 merely consists of a handful of XyWrite change commands, to be executed against a XYENCoded file.
The QDF1.PM XPL program does a different kind of cooking of XYENC output. Specifically, it adds whitespace to an XPL program, while attempting to use line breaks and indentation to reflect the logical structure of the XPL program. QDF1 assumes that it is working on a raw XYENC produced file that has not been significantly altered, so it should generally be used immediately after XYENCoding a file.
QDF1.PM is very much a heuristic program that can fail miserably if the input file doesn't conform to its assumptions. So you should inspect its work before accepting it. Should it fail miserably, adding suitable whitespace “by hand” is usually not out of the question.
QDF3.PM does yet another variety of cooking. It scans an encoded file and adds embedded XyWrite MD commands to identify primitive functions and XyWrite IV's 0FFh,0Cxh,0xxh specially encoded “search argument” characters. As provided, these are marked with MD13 and MD78 respectively, but the user can easily change this to match his preferences. QDF3 is very simple, only marking primitives that begin with a letter (e.g., it won't mark #3 ,@F ,&H , or $M primitives, for example), so you might want to “touch up” the work it does for an XPL program that includes some of the less used primitives. Also, in its MD78 marking of the special search characters, QDF3 assumes that all such characters have already been cooked to a form requiring only a single screen character for display -- if that is not the case for the file you apply it to, you also may have to do a bit of touch up.
QDM2 and QDM3 are both extremely simple XPL programs, and you may want to expand them or otherwise adjust them to your needs.
Although XyWrite always places an EOF character (a 1Ah byte, also sometime known as Control-Z) at the end of a files that it writes, that practice is now generally obsolete, and is mostly a nuisance. For example, if you use XyWrite to edit an HTML WEB page, you must take steps (utility programs exist for exactly this purpose) to remove the EOF character before the page will display properly using a WEB browser. In the modern world, then, many files that might be considered as XyWrite editable or (at some time) XyWrite edited files may, by deliberate intention, not be terminated with an EOF character.
XYDEC, in reconstructing a “XyWrite file” from an encoded one, does not automatically add an EOF character to the output file. If the original input file to XYENC had an EOF character, the encoded file will contain a (visible) “˜Z” (tilde,Z) token at the end, which reflects the existence of the EOF character in the original file. If that was the case, XYDEC will restore the EOF character when it restores the file. But if the original file had no EOF character, there will be no ˜Z token in the encoding, and no EOF character will be added to the restored file.
In short, with respect to EOF characters in XYDEC output files, XYDEC does exactly what it is told to do by the existence, or absence, of ˜Z (or equivalent) tokens in the encoded file. This includes putting EOF characters in the middle of the output file, if that is what the encoding says should be done. Management of EOF characters is in no way “automatic.”
If you are careless, this can lead to problems. Suppose we have two files, A.TXT and B.TXT, each with an ending EOF character. If I edit B.TXT with XyWrite, merge A.TXT in at the top of B.TXT (using XyWrite's MERGE command), and store the result, I will end up with a combined file that (still) has only one EOF character, at the end.
But suppose I XYENCode A.TXT to A.ENC, B.TXT to B.ENC. Then suppose I edit B.ENC with XyWrite, merge A.ENC in at the top, and save the result. If I then XYDECode the result, it will have two EOF characters in the result, and one of them will be in the middle of the file (where A.TXT ends). When subsequently loaded with XyWrite, the B.TXT portion will not be loaded.
This is because, when I directly merged A.TXT into B.TXT with XyWrite, XyWrite has a hidden behavior which “automatically” discards the EOF character from A.TXT. But when I merge A.ENC into B.ENC, the EOF on the A.TXT file is now represented as normal, visible, data, in the form of a “˜Z” token. Merging the files under XyWrite does not delete that token.
Also worth noting: XyWrite III+ does not require an EOF at the end of a file to be edited, and will be perfectly happy without one. But the same cannot be said for XyWrite III+ XPL programs to be loaded with the LDPM command -- removing the EOF on XPL programs to be loaded with LDPM can be disastrous. The actual requirement here is not the EOF character itself, but the fact that a XyWrite III+ XPL program can not end with a right guillemet -- if it does, the last top level embedded command will not be recognized as such and will be treated as data, rather than being executed. This is the same bug that caused XyWrite III+ XPL programmers to usually add a NO (No-Operation) primitive to the end of XPL routines loaded with the «SU» command -- but the bug was much more visible in the «SU» case, because the default EOF placed on XPL programs in file form pretty much hid the problem in the LDPM case.
XyWrite IV does not have this bug.
So, to summarize: EOF character creation, and EOF character deletion, is not, and can not be, “automatic” when working with the XYENCoded files, which are subsequently XYDECoded. If you merging or otherwise combining XYENCoded files or portions thereof, you must be careful to consider the presence or absence of ˜Z EOF tokens, and remove or add such tokens as required to get the EOF behavior that you actually want.
XYENC.COM
Usage: XYENC file_in file_out [nn] [L] [H] [B] [C]
Option switches:
nn : max output line length to be as indicated (20 to 99)
L : allow most Low chars (01h-1Fh) to pass without encoding
H : allow most High chars (80h-0FCh) to pass without encoding
B : do Both L and H
C : output to include CRC32 checksum token for input file
Specify filename of '.' for std input or std output
This is the encoding program. Most of what's here has been discussed. Note that the program normally works with files, but will accept redirected input from "Standard In" if "." is specified as the input filename. An output filename of "." similarly allows output redirection to "Standard Out."
When input is from redirection, XYENC does not put a timestamp token in the output file
When the output is redirected, the 'C' option will not work properly, and should not be specified.
When the 'nn' option is specified, in addition to breaking the output into lines, XYENC will add a "%" at the beginning of any line that would otherwise have started with a "." (period). This does not affect the decoding, and is done because lines beginning with a period are sometimes not transmitted properly by email systems.
When the 'C' option is specified, a CRC32 "CRC check" of the input file is calculated, and token reflecting that calculation is placed at the beginning of the output file. This CRC32 check value is compatible with the one generated by zip programs.
Although intended for, and most efficient with, XyWrite files, XYENC will encode any file.
XYENC will not overwrite an existing output file, unless the existing file has length zero.
XYDEC.COM
Usage: XYDEC file_in file_out
Specify filename of '.' for std input or std output
This is the decoding program.
Unlike input to XYENC, which will accept any input, it is possible for the input to XYDEC to contain what are effectively "syntax errors." When XYDEC encounters a token that it cannot decode, it copies the offending token to the output file, but brackets the token with "[[[[" in front, and "]]]]" in back. When this occurs, the program will signal an error at completion, and you should then search the output file for "[[[[" or "]]]]" to find the offending token(s).
If a checksum token is recognized in the input file, XYDEC computes the CRC32 check value of the output file it generates, and compares that value with the value in the checksum token, signalling an error if they disagree. This provides a quick mechanism to verify that the entire XYENC input, possible cooking, and XYDEC output cycle did not corrupt the original file. If you make intentionally make alterations on an encoded file that you expect to affect the resulting decoded file, you should delete the checksum token (if present) from the encoded file.
A checksum token is recognize only near the beginning of the encoded input file -- before any tokens or data are seen that generated decoded output.
If one or more timestamp tokens is found in the input file, the output file will be given the timestamp represented by the last such token encountered in the input file.
XYDEC will not overwrite an existing output file, unless the existing file has length zero.
The XPL programs provided in the package are as much "proof of concept" exercises as they are actual mature programs. I haven't really had a chance to use them much at all, and they may evolve considerably over time. This is not to say that they are broken or useless, but they almost certainly lack the sophistication that will come with some field use, feedback, and evolution.
QDF1.PM
is an XyWrite III+ compatible XPL program normally executed against a raw encoded file. QDF1 adds whitespace to the file, dividing the file into lines and indenting lines in an attempt to help show program structure. QDF1 makes assumptions about its input, and may fail miserably, so you inspect its output before accepting it.
Other XPL programs utilizing the somewhat richer XPL facilities of XyWrite IV may exist at locations such as www.serve.com/xywwwweb, which perform similarly. If you are a XyWrite IV user, and if these serve your needs better than QDF1, then you should use these instead of QDF1.
Relative to (my understanding of and guesses about) other XPL programs that do similar things, you should be aware that QDF1 makes use the implicit guillemet counting and "matching" that XyWrite does when it encloses even a deeply nested set of embedded commands in a single "formatting triangle." Hence, QDF1's formatting of output may be affected by unbalanced guillemets in the input file in ways that are considerably different that similar programs, and this may (or may not) turn out to be a useful property, relative to the problem of discovering unbalanced guillemets in an XPL program.
QDF2.PM
is a simple XyWrite III+ compatible XPL program that "cooks" many XYENC 2 byte encodings into XyWrite III+ style 3 byte encodings (which subsequently display as the original, single character under XyWrite).
QDF3.PM
is a simple XyWrite III+ compatible XPL programs that adds XyWrite MD commands to visually "mark" most primitives and XyWrite IV special "search argument encodings" in an encoded file.