User Tools

Site Tools


detokeniser

This is an old revision of the document!


Detokeniser

by Jon Ripley, January 2008, May 2011

The FNdetokenise function listed below accepts a string containing either a tokenised program line or tokenised code and returns a string containing detokenised code:

      DEF FNdetokenise(S$)
      IF S$ = "" OR S$ = CHR$0+CHR$255+CHR$255 THEN =""
      PRIVATE Keywd$()
      IF !^Keywd$()=0 THEN
        DIM Keywd$(160)
        Keywd$() = "AND","DIV","EOR","MOD","OR","ERROR","LINE","OFF",\
        \"STEP","SPC","TAB(","ELSE","THEN","","OPENIN","PTR",\
        \"PAGE","TIME","LOMEM","HIMEM","ABS","ACS","ADVAL","ASC",\
        \"ASN","ATN","BGET","COS","COUNT","DEG","ERL","ERR",\
        \"EVAL","EXP","EXT","FALSE","FN","GET","INKEY","INSTR(",\
        \"INT","LEN","LN","LOG","NOT","OPENUP","OPENOUT","PI",\
        \"POINT(","POS","RAD","RND","SGN","SIN","SQR","TAN",\
        \"TO","TRUE","USR","VAL","VPOS","CHR$","GET$","INKEY$",\
        \"LEFT$(","MID$(","RIGHT$(","STR$","STRING$(","EOF","SUM","WHILE",\
        \"CASE","WHEN","OF","ENDCASE","OTHERWISE","ENDIF","ENDWHILE","PTR",\
        \"PAGE","TIME","LOMEM","HIMEM","SOUND","BPUT","CALL","CHAIN",\
        \"CLEAR","CLOSE","CLG","CLS","DATA","DEF","DIM","DRAW",\
        \"END","ENDPROC","ENVELOPE","FOR","GOSUB","GOTO","GCOL","IF",\
        \"INPUT","LET","LOCAL","MODE","MOVE","NEXT","ON","VDU",\
        \"PLOT","PRINT","PROC","READ","REM","REPEAT","REPORT","RESTORE",\
        \"RETURN","RUN","STOP","COLOUR","TRACE","UNTIL","WIDTH","OSCLI",\
        \"","CIRCLE","ELLIPSE","FILL","MOUSE","ORIGIN","QUIT","RECTANGLE",\
        \"SWAP","SYS","TINT","WAIT","INSTALL","","PRIVATE","BY","EXIT"
      ENDIF
      LOCAL flag%, O$, i%, ch%
      IF ASCS$ = LENS$ AND ASCRIGHT$(S$,1) = 13 THEN
        ch% = ASC MID$(S$, 2)+256*ASC MID$(S$, 3)
        IF ch% O$=STR$ch%+" "
        S$ = MID$(S$, 4, LENS$-4)
        IF S$="" THEN =LEFT$(O$, LENO$-1)
      ENDIF
      FOR i%=1 TO LENS$
        IF ASCMID$(S$,i%)=34 flag%EOR=1
        IF flag% AND 1 THEN
          O$ += MID$(S$, i%, 1)
        ELSE
          ch% = ASCMID$(S$, i%)
          CASE TRUE OF
            WHEN ch% >= 17 AND ch% < 128:O$ += CHR$ch%
            WHEN ch% = &8D
              ch% = ASCMID$(S$,i%+1)
              lo% = ((ch% << 2) AND &C0) EOR ASC MID$(S$, i%+2)
              hi% = ((ch% << 4) AND &C0) EOR ASC MID$(S$, i%+3)
              O$+ = STR$(lo% + 256*hi%)
              i% += 3
            OTHERWISE O$ += Keywd$(ch% EOR 128)
          ENDCASE
        ENDIF
      NEXT i%
      =O$


Using FNdetokenise to list the error line

Using FNdetokeniser to write a Quine


Using FNdetokenise it is possible to write a Quine, that is, a program which when run outputs its own listing:

      P% = PAGE
      WHILE (!P% AND &FFFFFF) <> &FFFF00
        line$ = ""
        FOR i% = P% TO P%+?P%-1
          line$ += CHR$?i%
        NEXT i%
        PRINT FNdetokenise(line$)
        P% += ?P%
      ENDWHILE
      END

In the author's opinion using this method or *LIST to write a Quine is cheating.

Using the built-in detokeniser


The BBC BASIC for Windows interpreter has a built-in detokeniser accessed using the *LIST command that detokenises a specified file and outputs the listing on the current output device. Whilst it is possible to write a simpler looking detokeniser utilising *LIST the resulting routine is significantly slower due to the number of file operations required.

      DEF FNdetokenise(S$)
      IF S$="" OR S$=CHR$0+CHR$255+CHR$255 THEN =""
      IF LENS$>251 ERROR 100,"Line too long"
      LOCAL T%, fIn$, hFile%, hOut%
      DIM T% LOCAL 255
      SYS "GetTempPath", 256, T%
      SYS "GetTempFileName", $$T%, "tmp", 0, T%:fIn$=$$T%
      SYS "GetTempPath", 256, T%
      SYS "GetTempFileName", $$T%, "tmp", 0, T%
      hFile% = OPENOUT$$T%:IF hFile% = 0 ERROR 100,"Cannot create temp file"
      IF ASCS$=LENS$ AND ASC RIGHT$(S$,1)=13 THEN
        BPUT#hFile%, S$+CHR$0+CHR$255+CHR$255;
      ELSE
        BPUT#hFile%, CHR$(LEN(S$)+4)+CHR$0+CHR$0+S$+CHR$13+CHR$0+CHR$255+CHR$255;
      ENDIF
      CLOSE#hFile%
      hOut% = @vdu%?-130
      hFile%=OPENUPfIn$:IF hFile%=0 ERROR 100,"Cannot create temp file"
      OSCLI "Output "+STR$hFile%
      OSCLI "List """+$$T%+""""
      OSCLI "Output "+STR$hOut%
      PTR#hFile%=0:S$=GET$#hFile%:CLOSE#hFile%
      WHILE ASCS$=32:S$=MID$(S$,2):ENDWHILE
      SYS "DeleteFile", fIn$
      SYS "DeleteFile", $$T%
      =S$



Do not use this version in production code.

This website uses cookies for visitor traffic analysis. By using the website, you agree with storing the cookies on your computer.More information
detokeniser.1522502355.txt.gz · Last modified: 2018/03/31 13:19 by 127.0.0.1