• DDMsgReader doesn't display certain encoded characters properly

    From Eric Oulashin@1:103/705 to GitLab issue in main/sbbs on Mon Apr 3 14:18:44 2023
    open https://gitlab.synchro.net/main/sbbs/-/issues/541

    Some characters in messages (such as the apostrophe used apparently by Apple Mail or other software on iPads) don't display properly in DDMsgReader (and show up as odd-looking characters). This may be related to MIME encoding? msglist.js seems to display them properly, so maybe that can be used as an example.Some notes on dumping a message, which may help in debugging:<DigitalMan> 'smbutil v /sbbs/data/mail' will dump all the headers<DigitalMan> 'smbutil r /sbbs/data/mail' will read the message text<DigitalMan> if you know the message number, you can skip all the other messages easily, or just pipe output to a file and edit to suit<nelgin> I did, thank you. uploaded bademail.txt and badheader.txt and I'm using SyncTerm, of course :)<nelgin> The sender was using MuffinTerm on an iphone fwiw.<DigitalMan> the terminals (on the sending or receiving side) shouldn't really matter<DigitalMan> the sender was using Apple Mail (2.3445.9.7) according to those headers<git_nelgin> GitLab issue from Nigel Reed<git_nelgin> open https://gitlab.synchro.net/main/sbbs/-/issues/540<git_nelgin> add ssh passphrase authentication<nelgin> Ah right, he was using muffinterm to access the bbs but yeah, that was via gmail. My bad.<DigitalMan> Can you try viewing that same message using 'smbutil -r r /sbbs/data/mail' ? Does it look the same or add a like of MIME noise to the output?<DigitalMan> -r = display raw message body text (not MIME-decoded)One of DM's recent commits:git_rswindell> https://gitlab.synchro.net/main/sbbs/-/commit/1ca7f708fe87ea825cb0e5af<git_rswindell> Modified src/smblib/smbtxt.c<git_rswindell> Fix decoding of tab-indented 'charset' of multi-part MIME text parts<git_rswindell> Apple Mail apparently uses tabs to indent the charset of nested MIME parts:
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Mon Apr 3 14:46:46 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3414

    Does DDMsgReader translate UTF-8 encoded messages to CP437 or ASCII when appropriate for the viewing terminal?
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Eric Oulashin@1:103/705 to GitLab note in main/sbbs on Mon Apr 3 17:33:14 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3420

    I did add some code quite a while ago that used utf8_cp437() (from utf8_cp437.js) for that purpose, but it seems that wasn't enough. I hadn't gotten back to that until today.I had a look at what msglist.js is doing, and it has the function msg_pmode(). It looks like if the message header's is_utf8 property is true, then it sets the P_UTF8 mode bit, and it looks like that's required to pass to console.print() in order for it to look valid on a terminal.
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Mon Apr 3 18:00:59 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3421

    Replace "a terminal" with "a non-UTF-8" terminal and yeah, that'd be true. Some terminals support UTF-8 just fine (and in those cases, don't support CP437).
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab issue in main/sbbs on Tue Apr 4 13:37:29 2023
    close https://gitlab.synchro.net/main/sbbs/-/issues/541
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Nigel Reed@1:103/705 to GitLab issue in main/sbbs on Wed Apr 5 01:15:19 2023
    reopen https://gitlab.synchro.net/main/sbbs/-/issues/541

    Some characters in messages (such as the apostrophe used apparently by Apple Mail or other software on iPads) don't display properly in DDMsgReader (and show up as odd-looking characters). This may be related to MIME encoding? msglist.js seems to display them properly, so maybe that can be used as an example.Some notes on dumping a message, which may help in debugging:```<DigitalMan> 'smbutil v /sbbs/data/mail' will dump all the headers<DigitalMan> 'smbutil r /sbbs/data/mail' will read the message text<DigitalMan> if you know the message number, you can skip all the other messages easily, or just pipe output to a file and edit to suit<nelgin> I did, thank you. uploaded bademail.txt and badheader.txt and I'm using SyncTerm, of course :)<nelgin> The sender was using MuffinTerm on an iphone fwiw.<DigitalMan> the terminals (on the sending or receiving side) shouldn't really matter<DigitalMan> the sender was using Apple Mail (2.3445.9.7) according to those headers<nelgin> Ah right, he was using muffinterm to access the bbs but yeah, that was via gmail. My bad.<DigitalMan> Can you try viewing that same message using 'smbutil -r r /sbbs/data/mail' ? Does it look the same or add a like of MIME noise to the output?<DigitalMan> -r = display raw message body text (not MIME-decoded)```One of DM's recent commits:```<git_rswindell> https://gitlab.synchro.net/main/sbbs/-/commit/1ca7f708fe87ea825cb0e5af<git_rswindell> Modified src/smblib/smbtxt.c<git_rswindell> Fix decoding of tab-indented 'charset' of multi-part MIME text parts<git_rswindell> Apple Mail apparently uses tabs to indent the charset of nested MIME parts:```
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Nigel Reed@1:103/705 to GitLab note in main/sbbs on Wed Apr 5 01:15:30 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3437

    Issue not fixed by latest commits.
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Eric Oulashin@1:103/705 to GitLab note in main/sbbs on Tue May 2 13:19:54 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3552

    Have you found anything new regarding this issue? I'm wondering if this can be closed.
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Eric Oulashin@1:103/705 to GitLab note in main/sbbs on Tue May 2 13:32:29 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3553

    Closing, as this appears to be fixed and there's no new information for this issue.
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)