• Compiling from source

    From Avon@21:1/101 to All on Fri Jan 15 22:51:38 2021
    I'm trying to compile some software from source and hitting errors. It seems like it's missing files and I think some of the issue was the compile files were looking for lower case named files but the zip of the source code
    had files saved in upper case. I changed the files to lowercase then ran this best guess.

    Note, I've not used gcc much at all.

    gcc config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp socket.cpp socket.hpp sock.h spoon.cpp spoon.h -o spoon.test

    The docs say the software is meant to compile using gcc for Linux, you can
    see the files have .cpp extensions as well as .h and .hpp

    The docs say there should be a suggested make shell script for gcc with it,
    but this seems to be missing in the zip.

    The other associated programs have the shell scripts but why the one bit of software I am trying to build is missing it, is anyones guess.

    I attempted my above command line based on some other examples where there
    was info on how to call gcc... that said I don't know which files to include
    in the build and which to leave out.

    When I ran this I got the following

    In file included from spoon.h:115,
    from config.cpp:38:
    socket.hpp:73:10: fatal error: iostream.h: No such file or directory
    #include <iostream.h>
    ^~~~~~~~~~~~
    compilation terminated.
    In file included from spoon.h:115,
    from keyboard.cpp:38:
    socket.hpp:73:10: fatal error: iostream.h: No such file or directory
    #include <iostream.h>
    ^~~~~~~~~~~~
    compilation terminated.
    In file included from spoon.h:115,
    from nntp.cpp:38:
    socket.hpp:73:10: fatal error: iostream.h: No such file or directory
    #include <iostream.h>
    ^~~~~~~~~~~~
    compilation terminated.
    In file included from spoon.h:115,
    from pop.cpp:38:
    socket.hpp:73:10: fatal error: iostream.h: No such file or directory
    #include <iostream.h>
    ^~~~~~~~~~~~
    compilation terminated.
    In file included from spoon.h:115,
    from smtp.cpp:38:
    socket.hpp:73:10: fatal error: iostream.h: No such file or directory
    #include <iostream.h>
    ^~~~~~~~~~~~
    compilation terminated.
    In file included from socket.cpp:8:
    socket.hpp:73:10: fatal error: iostream.h: No such file or directory
    #include <iostream.h>
    ^~~~~~~~~~~~
    compilation terminated.
    socket.hpp:73:10: fatal error: iostream.h: No such file or directory
    #include <iostream.h>
    ^~~~~~~~~~~~
    compilation terminated.
    In file included from spoon.h:115,
    from spoon.cpp:40:
    socket.hpp:73:10: fatal error: iostream.h: No such file or directory
    #include <iostream.h>
    ^~~~~~~~~~~~
    compilation terminated.
    In file included from spoon.h:115:
    socket.hpp:51:8: error: expected identifier or \u2018(\u2019 before string constant
    extern "C" {
    ^~~
    In file included from spoon.h:115:
    socket.hpp:73:10: fatal error: iostream.h: No such file or directory
    #include <iostream.h>
    ^~~~~~~~~~~~
    compilation terminated.

    iostream.h does not appear to be in the zip file with this source code, could it be some other known file I can source elsewhere?

    If there's anyone about that has some experience in these matters or could
    help getting this complied, please sing out.

    Thanks :)

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From apam@21:1/182 to Avon on Fri Jan 15 20:03:25 2021
    gcc config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp socket.cpp
    socket.hpp
    sock.h spoon.cpp spoon.h -o spoon.test

    Given they are c++ files, not plain c, you might have better luck with g++

    g++ is the C++ compiler, gcc is C compiler.

    Andrew


    --- Talisman v0.10-dev (Linux/armv7l)
    * Origin: HappyLand v2.0 - telnet://happyland.zapto.org:11892/ (21:1/182)
  • From Oli@21:3/102 to Avon on Fri Jan 15 12:57:18 2021
    Avon wrote (2021-01-15):

    I'm trying to compile some software from source and hitting errors. It seems like it's missing files and I think some of the issue was the
    compile files were looking for lower case named files but the zip of the source code had files saved in upper case. I changed the files to
    lowercase then ran this best guess.

    Note, I've not used gcc much at all.

    gcc config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp socket.cpp
    socket.hpp sock.h spoon.cpp spoon.h -o spoon.test

    soupgate?

    I succesfully compiled it on Linux some years ago. Does it work with g++ as apam mentioned. If not I have it zipped as a backup and can look if I made any changes to get it compile. I might discovered some bug, I'm not sure anyomre if it was soupgate or some other software I used with soupgate or just the pktview tool.

    ---
    * Origin: this message must NOT be gated to Telegram (21:3/102)
  • From Avon@21:1/101 to Oli on Sat Jan 16 09:40:34 2021
    On 15 Jan 2021 at 12:57p, Oli pondered and said...

    soupgate?

    I succesfully compiled it on Linux some years ago. Does it work with g++ as apam mentioned. If not I have it zipped as a backup and can look if I made any changes to get it compile. I might discovered some bug, I'm not sure anyomre if it was soupgate or some other software I used with soupgate or just the pktview tool.

    Soup, The version that's avail from the website is not the latest it seems
    and missing some authenticated login to NNTP that I'm after :(
    Thanks for the info and offer :)

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to apam on Sat Jan 16 09:44:26 2021
    On 15 Jan 2021 at 08:03p, apam pondered and said...

    Given they are c++ files, not plain c, you might have better luck with
    g++
    g++ is the C++ compiler, gcc is C compiler.

    Sadly nope, still missing some files called in one of the other files I think.

    Oli has provided me with a possible lead so just chasing that up now :)

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Sat Jan 16 11:48:27 2021
    Avon wrote (2021-01-16):

    On 15 Jan 2021 at 12:57p, Oli pondered and said...

    soupgate?

    I succesfully compiled it on Linux some years ago. Does it work
    with g++ as apam mentioned. If not I have it zipped as a backup and
    can look if I made any changes to get it compile. I might
    discovered some bug, I'm not sure anyomre if it was soupgate or
    some other software I used with soupgate or just the pktview tool.

    Soup, The version that's avail from the website is not the latest it seems and missing some authenticated login to NNTP that I'm after :(
    Thanks for the info and offer :)

    So spoon would be the replacement for VSoup? It can feed/create soup packets to/from nntp and smtp/pop3 servers?

    Is there a NNTP server that serves these gated messages (from FSXnet?) and can I (anyone) access it? I'm interested how well the gating works.

    ---
    * Origin: this message must NOT be gated to Telegram (21:3/102)
  • From Avon@21:1/101 to Oli on Sun Jan 17 09:49:18 2021
    On 16 Jan 2021 at 11:48a, Oli pondered and said...

    So spoon would be the replacement for VSoup? It can feed/create soup packets to/from nntp and smtp/pop3 servers?

    Yes that's my understanding although I have yet to test all of this.

    Is there a NNTP server that serves these gated messages (from FSXnet?)
    and can I (anyone) access it? I'm interested how well the gating works.

    news.bbs.nz is the server I run that the gateway software at 1/10 connects with. The NNTP server does not (at this time) carry the fsxNet FTN echos.
    You should be able to poll it to obtain the usual big 8 Usenet
    newsgroups / hierarchies as read access is open to all. Post access requires authenticated login. There are a few groups like nz.test and agency.test you can use though to test against.

    Feel free to netmail me with any other questions :)

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From tenser@21:1/101 to Avon on Mon Jan 18 02:14:09 2021
    On 15 Jan 2021 at 10:51p, Avon pondered and said...

    gcc config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp socket.cpp socket.hpp sock.h spoon.cpp spoon.h -o spoon.test

    Generally speaking, you don't try to compile header files
    (.hpp or .h) directly. Rather, those are usually meant to
    be "included" in a source file that is then compiled.
    Header files (sometimes called "include files") usually
    contain things like structure and type definitions and
    function declarations that you want visible in more than
    one source file.

    Running `gcc` by itself may well be just fine, as it's
    generally smart enough to understand what language it is
    compiling based on the filename (specifically, the
    "extension", though one should be careful to note that
    Unix-like systems don't assign any special meaning to
    file extensions itself, unlike DOS/Windows/DEC systems;
    it leaves assigning meaning to parts of a filename to
    user-level programs...I digress). But running g++ or
    c++ is also fine and certainly won't hurt anything.

    The docs say there should be a suggested make shell script for gcc with it, but this seems to be missing in the zip.

    There is no `makefile` or `Makefile`? That's unfortunate.

    In file included from spoon.h:115,
    from config.cpp:38:
    socket.hpp:73:10: fatal error: iostream.h: No such file or directory
    #include <iostream.h>

    This is one of two things:

    1) Since the compiler was invoked as "gcc" and the ".h" file
    extension is usually associated with C language header files,
    it is trying to compile that file in C mode. However, as
    mentioned above, one doesn't generally compiler header
    directly; simply omit the ".h" and ".hpp" files from the command
    line and see if that helps?

    2) <iostream.h> has itself been deprecated for almost 25 years
    (since the first official C++ standard in 1998) and compilers
    are starting to ship without it. The replacement is <iostream>
    (without the ending ".h"), but you'll almost certainly have to
    add a, "using namespace std;" somewhere in source files that
    include it, as code that's old enough to use <iostream.h> is
    probably also old enough to predate the widespread use of C++
    namespaces.

    I'd try pulling the headers off of the command line first to
    see if that helps.

    If the tarball of the file in question is available somewhere,
    I might be able to pull of a few cycles to see if I can get it
    to compile.

    Hmm. There really ought to be a Github organization or
    something for ancient BBS software updated to compile on
    modern systems.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to tenser on Mon Jan 18 10:43:30 2021
    tenser wrote (2021-01-18):

    The docs say there should be a suggested make shell script for gcc
    with it, but this seems to be missing in the zip.

    There is no `makefile` or `Makefile`? That's unfortunate.

    makespoon which is just

    g++ spoon.cpp config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp socket.cpp -o spoon

    2) <iostream.h> has itself been deprecated for almost 25 years
    (since the first official C++ standard in 1998) and compilers
    are starting to ship without it. The replacement is <iostream>
    (without the ending ".h"), but you'll almost certainly have to
    add a, "using namespace std;" somewhere in source files that
    include it, as code that's old enough to use <iostream.h> is
    probably also old enough to predate the widespread use of C++
    namespaces.

    I think this is what I did some years ago. See https://gitlab.com/fidosoft/soupgate

    I still get warnings like:

    $ ./makespoon
    spoon.cpp:60:7: warning: built-in function ‘logf’ declared as non-function [-Wbuiltin-declaration-mismatch]
    FILE *logf = NULL;
    ^~~~
    spoon.cpp: In function ‘void checkdir(char*)’:
    spoon.cpp:143:69: warning: ISO C++ forbids converting a string constant to ‘char*’ [-Wwrite-strings]
    logprintf("FATAL ERROR: subdirectory %s does not exist\n", dir);

    but it compiles and run.

    Hmm. There really ought to be a Github organization or
    something for ancient BBS software updated to compile on
    modern systems.

    That's a good idea

    There is some software at https://github.com/ftnapps (which should compile)

    some I collected at https://github.com/fidosoft and https://gitlab.com/fidosoft, but many don't compile.

    It would be good to have people available who can help to get things build again. I think most of the time the errors are not very hard to fix, if you know C(++) and know what's changed over the years. For people without C experience (like me) it's sometimes hard to figure out how to fix it properly without introducing new bugs.

    Are there any limitations for organizations on the free github plan or is it good enough for public repos? (I only remember that I could do more with gitlab some years ago, but this was before Microsoft took over and was maybe related to private repos).

    ---
    * Origin: this message must NOT be gated to Telegram (21:3/102)
  • From tenser@21:1/101 to Oli on Tue Jan 19 03:20:45 2021
    On 18 Jan 2021 at 10:43a, Oli pondered and said...

    makespoon which is just

    g++ spoon.cpp config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp
    socket.cpp -o spoon

    I see; they distributed it as a shell script. It seems like
    that may be missing from Paul's copy of the archive, though.

    I think this is what I did some years ago. See https://gitlab.com/fidosoft/soupgate

    I still get warnings like:

    $ ./makespoon
    spoon.cpp:60:7: warning: built-in function ‘logf’ declared as non-function [-Wbuiltin-declaration-mismatch]
    FILE *logf = NULL;
    ^~~~

    Ah, this is due to an unfortunate name. `logf` is the
    name of a function from the standard math library
    (single-precision floating point natural logarithm),
    and as per the standard, names of standard functions
    are reserved. That's also untenable for names like
    `log`, hence why this is a warning, not an error.
    Simply renaming the variable would eliminate that,
    though.

    spoon.cpp: In function ‘void checkdir(char*)’:
    spoon.cpp:143:69: warning: ISO C++ forbids converting a string constant
    to ‘char*’ [-Wwrite-strings]
    logprintf("FATAL ERROR: subdirectory %s does not exist\n", dir);

    but it compiles and run.

    This is a single type error. `logprintf` is probably
    defined to take `char *` as its first argument; this
    warning is saying that one is passing a string constant
    as that argument, but that the function is defined so
    that it could mutate the thing it points to. The fix
    here is to add a `const` qualifier to the argument.

    It would be good to have people available who can help to get things
    build again. I think most of the time the errors are not very hard to
    fix, if you know C(++) and know what's changed over the years. For
    people without C experience (like me) it's sometimes hard to figure out how to fix it properly without introducing new bugs.

    Yeah, most of this stuff is pretty simple. A lot of it
    is just that the code predates the relevant standards, or
    has obvious bugs.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to tenser on Mon Jan 18 16:49:55 2021
    tenser wrote (2021-01-19):

    I still get warnings like:

    $ ./makespoon
    spoon.cpp:60:7: warning: built-in function ‘logf’ declared as
    non-function [-Wbuiltin-declaration-mismatch]
    FILE *logf = NULL;
    ^~~~

    Ah, this is due to an unfortunate name. `logf` is the
    name of a function from the standard math library
    (single-precision floating point natural logarithm),

    that worked.

    spoon.cpp: In function ‘void checkdir(char*)’:
    spoon.cpp:143:69: warning: ISO C++ forbids converting a string
    constant to ‘char*’ [-Wwrite-strings]
    logprintf("FATAL ERROR: subdirectory %s does not exist\n",
    dir);

    but it compiles and run.

    This is a single type error. `logprintf` is probably
    defined to take `char *` as its first argument; this
    warning is saying that one is passing a string constant
    as that argument, but that the function is defined so
    that it could mutate the thing it points to. The fix
    here is to add a `const` qualifier to the argument.

    void logprintf(const char *s, ...);

    compiles, but so does

    void logprintf(char const *s, ...);

    is it the same?

    ---
    * Origin: this message must NOT be gated to Telegram (21:3/102)
  • From tenser@21:1/101 to Oli on Tue Jan 19 05:48:30 2021
    On 18 Jan 2021 at 04:49p, Oli pondered and said...

    void logprintf(const char *s, ...);

    compiles, but so does

    void logprintf(char const *s, ...);

    Yup. Those are the same.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to tenser on Thu Jan 21 20:21:16 2021
    On 18 Jan 2021 at 02:14a, tenser pondered and said...

    gcc config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp socket.cpp socket.hpp sock.h spoon.cpp spoon.h -o spoon.test

    Generally speaking, you don't try to compile header files
    (.hpp or .h) directly. Rather, those are usually meant to
    be "included" in a source file that is then compiled.

    Thanks, yep even to this untrained bloke it seemed like when I looked at the errors there were things being called that were missing. I had a feeling not all files needed to be included in the command line but also no idea of what
    I should or should not use. :)

    Running `gcc` by itself may well be just fine, as it's

    user-level programs...I digress). But running g++ or
    c++ is also fine and certainly won't hurt anything.

    Yep I tried both but no joy.

    There is no `makefile` or `Makefile`? That's unfortunate.

    No the docs said there would be but no dice.

    I'd try pulling the headers off of the command line first to
    see if that helps.

    If the tarball of the file in question is available somewhere,
    I might be able to pull of a few cycles to see if I can get it
    to compile.

    I have not looked / played with it in a few days now but thanks for the offer.

    I did try a version that Oli passed to me via a git repo but I have not yet
    had success getting soupgate and soup to work totally correctly as yet for
    any static or complied versions I have managed to build or obtain as yet.

    http://software.tomsweb.net/soupgate.html has the links to the static files
    and linux source.

    So far under linux (and I'm a bit woolly on this now as it's been a few days)

    I have managed to get (I think it was Oli's ) version of Soup to poll my NNTP server and pull down some packets, then I think it was the static soupgate
    file that gated them to PKT, then (and lordy I have got myself a bit muddled now with all the versions I have been trying) it was Oli's soupgate that
    would at least see the PKT to be exported in the FTN outbound, the static version didn't see them, and I (only once) got I think his version of soup to post to NNTP but then it borked after the NNTP bit as files on the HDD that should have been removed were not. Phew..

    You'll understand why I have not been back at this in few days.

    At the heart of all this is my need to find Linux alternatives to what I currently run on Win32 to parse FTN PKTs in/out of my NNTP server. For
    windows I have been using vSoup but that does not exist for Linux as far as I can tell hence playing with the authors Soup software to see if that would work.

    I may to start with end up having to fudge things between two systems linux
    and windows in order to get gating working while starting to run Linux for
    HUB packet tossing duties.

    To start with I plan to cut over things that handle my Fido and othernets traffic that I carry but not fsxNet as yet... I really want to sort things
    out using the othernets first before I look to move 21:1/100 over as well..

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From deon@21:2/116 to Avon on Thu Jan 21 22:24:11 2021
    Re: Re: Compiling from source
    By: Avon to tenser on Thu Jan 21 2021 08:21 pm

    Hey,

    At the heart of all this is my need to find Linux alternatives to what I currently run on Win32 to parse FTN PKTs in/out of
    my NNTP server. For

    A thought - you should be able to get Syncronet being your NNTP to FTN gateway. It works well (I'm using it to get some newsgroups from eternal september).

    I'm already feeding a couple of folks with it (can feed anybody else if there is interest).

    ...ëîåï

    ... A triangle which has an angle of 135 degrees is called an obscene triangle --- SBBSecho 3.11-Linux
    * Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)
  • From Avon@21:1/101 to deon on Fri Jan 22 12:27:05 2021
    On 21 Jan 2021 at 10:24p, deon pondered and said...

    A thought - you should be able to get Syncronet being your NNTP to FTN gateway.

    It works well (I'm using it to get some newsgroups from eternal september).

    OK thanks, yeah I'll keep it up my sleeve as a plan B, C or D :)

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Fri Jan 22 08:35:02 2021
    Avon wrote (2021-01-22):

    On 21 Jan 2021 at 10:24p, deon pondered and said...

    A thought - you should be able to get Syncronet being your NNTP to
    FTN gateway.

    It works well (I'm using it to get some newsgroups from eternal
    september).

    OK thanks, yeah I'll keep it up my sleeve as a plan B, C or D :)

    There is also ifgate, fidogate, watergate as plan E, F, ...

    ---
    * Origin: this message must NOT be gated to Telegram (21:3/102)
  • From Avon@21:1/101 to Oli on Fri Jan 22 21:25:49 2021
    On 22 Jan 2021 at 08:35a, Oli pondered and said...

    There is also ifgate, fidogate, watergate as plan E, F, ...

    It might yet come to that. I wish I weren't so darn tired tonight.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Fri Jan 22 09:53:06 2021
    Avon wrote (2021-01-22):

    On 22 Jan 2021 at 08:35a, Oli pondered and said...

    There is also ifgate, fidogate, watergate as plan E, F, ...

    It might yet come to that.

    Just in case, there is a fidogate fork with regular updates on github: https://github.com/ykaliuta/fidogate

    I never tried it, but I guess if the simpler solutions don't work, it could be done with fidogate.

    Watergate looked really nice, but the last release is pretty old. It would be a good candidate for porting to a recent FreePascal / Linux though ;)

    I wish I weren't so darn tired tonight.

    I wish I weren't so tired this morning :)

    * Origin: this message must NOT be gated to Telegram (21:3/102)
  • From Avon@21:1/101 to Oli on Sat Jan 23 10:47:35 2021
    On 22 Jan 2021 at 09:53a, Oli pondered and said...

    Just in case, there is a fidogate fork with regular updates on github: https://github.com/ykaliuta/fidogate

    I've spent a further 3 hours today trying to get various versions of soupgate for linux to work correctly and also soup.

    What I found was

    oli soup - will compile and seems to happily pass login/pass to NNTP and then pull down / create soup packets for newsgroups.

    oli soupgate import failed with errors, saying it couldn't read the files it was trying to import

    I then used the static compiled soupgate.static from the website to run
    import and that worked to created FTN packets the HUB could import

    Testing the otherway from HUB to gate to NNTP has been the real issue.

    oli soupgate export worked but oli spoon failed to post to NNTP reporting issues in the files it was trying to post

    * Posting article to nz.test
    NNTP host returned error: 441 Can't parse Date: header
    ! NNTP server did not accept message
    free(): double free detected in tcache 2
    Aborted

    I reset the export test and used soupgate.static export but it failed to see the FTN packet in the outbound directory despite it being present :(

    On a hunch I changed the .clo file in the outbound to .CLO and then the soupgate.static found the packet and exported it.

    Then I tried using oli soup to send to NNTP and while it posted it still also fully failed to run correctly borking with this error

    * Posting article to nz.test
    free(): double free detected in tcache 2
    Aborted

    The REPLIES file in my Soup folder was still left behind, I think related to this abort.

    In sum,

    3 hours I can't get back :)
    Seems problematic to get two linux versions of files working.
    No idea why the static soupgate seems so picky about exporting files and only happy to look for upper case BSO files.. sigh.

    I may well soon turn attention to researching and trying to get another
    option working for linux.

    I want to get gateway stuff sorted before putting the Fido HUB with HPT
    online.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to Oli on Sat Jan 23 10:57:43 2021
    On 22 Jan 2021 at 09:53a, Oli pondered and said...

    Watergate looked really nice, but the last release is pretty old. It
    would be a

    where do I find this one?

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Sat Jan 23 10:02:05 2021
    Avon wrote (2021-01-23):

    I've spent a further 3 hours today trying to get various versions of soupgate for linux to work correctly and also soup.
    [...]
    In sum,

    3 hours I can't get back :)
    Seems problematic to get two linux versions of files working.

    Could it be some kind of 64-bit problem? If I find the time I will try soupgate on 32bit and 64bit.

    I may well soon turn attention to researching and trying to get another option working for linux.

    It's not unlikely that fidogate will get you a running and stable system faster than soupgate, even it takes some time to learn and configure it. But I have never used it, just guessing.

    I've installed Cyrus as an IMAP and NNTP server and I'm also interested to play around with FTN<->NNTP gating.

    ---
    * Origin: . (21:3/102)
  • From Oli@21:3/102 to Avon on Sat Jan 23 12:54:03 2021
    Avon wrote (2021-01-23):

    On 22 Jan 2021 at 09:53a, Oli pondered and said...

    Watergate looked really nice, but the last release is pretty old. It
    would be a

    where do I find this one?

    https://sourceforge.net/projects/watergate

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Sun Jan 24 11:19:53 2021
    On 23 Jan 2021 at 10:02a, Oli pondered and said...

    Could it be some kind of 64-bit problem? If I find the time I will try soupgate

    on 32bit and 64bit.

    I'm running Debian Buster X86_64 on this box.

    If there's something I need to do to compile the source differently or ? I'd
    be open to ideas. Thanks :)

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to Oli on Sun Jan 24 11:22:13 2021
    On 23 Jan 2021 at 10:02a, Oli pondered and said...

    It's not unlikely that fidogate will get you a running and stable system faster
    than soupgate, even it takes some time to learn and configure it. But I have never used it, just guessing.

    It seems like quite a beast in that it also offers not just gating but
    tossing etc. as well. I'm well down the path of setting up HPT as a tosser so not really wanting to use that aspect of Fidogate if I don't need to. But who knows.

    It's a bit disheartening trying to get what is a simple process involving two exe files in windows working on Linux.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Sun Jan 24 09:37:57 2021
    Avon wrote (2021-01-24):

    On 23 Jan 2021 at 10:02a, Oli pondered and said...

    It's not unlikely that fidogate will get you a running and stable
    system faster
    than soupgate, even it takes some time to learn and configure it.
    But I have never used it, just guessing.

    It seems like quite a beast in that it also offers not just gating but tossing etc. as well. I'm well down the path of setting up HPT as a
    tosser so not really wanting to use that aspect of Fidogate if I don't
    need to. But who knows.

    :)

    I think fidogate does not support message bases like Squish or JAM. Only pass-through tossing and gating to/from an NNTP server.

    It's a bit disheartening trying to get what is a simple process involving two exe files in windows working on Linux.

    Yeah ... FTN / BBS is really a niche and most software is unmaintained.

    ---
    * Origin: . (21:3/102)
  • From Oli@21:3/102 to Avon on Sun Jan 24 11:15:32 2021
    Avon wrote (2021-01-24):

    On 23 Jan 2021 at 10:02a, Oli pondered and said...

    Could it be some kind of 64-bit problem? If I find the time I will
    try soupgate

    on 32bit and 64bit.

    I'm running Debian Buster X86_64 on this box.

    If there's something I need to do to compile the source differently or ? I'd be open to ideas. Thanks :)

    there are a bunch of variables defined as "long", which I think are 64-bit integers on 64-bit Linux and 32-bit on 32-bit Linux. I'm not sure that it makes a difference for soupgate/spoon, but you could try the -m32 parameter with gcc/g++

    https://www.geeksforgeeks.org/compile-32-bit-program-64-bit-gcc-c-c/

    ---
    * Origin: . (21:3/102)
  • From Oli@21:3/102 to Oli on Sun Jan 24 14:25:03 2021
    Oli wrote (2021-01-24):

    on 32bit and 64bit.

    I'm running Debian Buster X86_64 on this box.

    If there's something I need to do to compile the source differently
    or ? I'd be open to ideas. Thanks :)

    there are a bunch of variables defined as "long", which I think are
    64-bit integers on 64-bit Linux and 32-bit on 32-bit Linux. I'm not sure that it makes a difference for soupgate/spoon, but you could try the -m32 parameter with gcc/g++

    found the first problem in pktview. It doesn't show the correct header correct in the 64-bit version. We should assume the 64-bit build is broken for soupgate and spoon too until otherwise proven.

    ---
    * Origin: . (21:3/102)
  • From Oli@21:3/102 to Oli on Sun Jan 24 14:29:19 2021
    Oli wrote (2021-01-24):

    found the first problem in pktview. It doesn't show the correct header correct in the 64-bit version. We should assume the 64-bit build is
    broken for soupgate and spoon too until otherwise proven.

    32-bit (Raspi) - correct
    --- PACKED MESSAGE #1 ---
    Version : 2
    Originating net : 2
    Originating node : 116
    Destination net : 3
    Destination node : 102
    Cost : 0
    Attributes : Pvt
    Date and time : 24 Jan 21 21:07:42

    64-bit (amd64) - all wrong
    --- PACKED MESSAGE #1 ---
    Version : 102
    Originating net : 1
    Originating node : 2
    Destination net : 0
    Destination node : 3
    Cost : 18976
    Attributes : Cra F/A I/T (?) Rrq Cpt
    Date and time : an 21 21:07:42

    ---
    * Origin: . (21:3/102)
  • From Oli@21:3/102 to Avon on Sun Jan 24 16:14:17 2021
    Avon wrote (2021-01-24):

    On 23 Jan 2021 at 10:02a, Oli pondered and said...

    Could it be some kind of 64-bit problem? If I find the time I will
    try soupgate

    on 32bit and 64bit.

    I'm running Debian Buster X86_64 on this box.

    If there's something I need to do to compile the source differently or ? I'd be open to ideas. Thanks :)

    could you test, if a 32-bit compile works? pktview does work form me with "gcc -m32" even on 64-bit Debian:

    apt install gcc-multilib g++-multilib

    g++ -m32 spoon.cpp config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp socket.cpp -o spoon
    gcc -m32 soupgate.c config.c lists.c pkt.c import.c export.c -o soupgate
    gcc -m32 pktview.c -o pktview

    If it does work, maybe we could port soupgate and tools to 64-bit.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Mon Jan 25 19:16:42 2021

    On 24 Jan 2021 at 04:14p, Oli pondered and said...

    could you test, if a 32-bit compile works? pktview does work form me
    with "gcc -m32" even on 64-bit Debian:

    Just so I'm clear, I need to be running a 32bit version of Linux to do the compile and testing?

    Or are you asking me to just run the compile commands on Debian 64bit and
    then see if it works?

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Mon Jan 25 07:43:12 2021
    Avon wrote (2021-01-25):

    On 24 Jan 2021 at 04:14p, Oli pondered and said...

    could you test, if a 32-bit compile works? pktview does work form me
    with "gcc -m32" even on 64-bit Debian:

    Just so I'm clear, I need to be running a 32bit version of Linux to do the compile and testing?

    no, you don't.

    Or are you asking me to just run the compile commands on Debian 64bit and then see if it works?

    yes

    apt install gcc-multilib g++-multilib

    installs the necessary 32-bit libraries (on a 64-bit system)

    "gcc -m32" compiles a 32-bit binary even if the OS is 64-bit.

    I can also send you the built binaries. They have been compiled on Debian Testing (Bullseye), but I think they should work.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Mon Jan 25 19:55:46 2021
    On 25 Jan 2021 at 07:43a, Oli pondered and said...

    Just so I'm clear, I need to be running a 32bit version of Linux to do compile and testing?

    no, you don't.

    10-4

    Or are you asking me to just run the compile commands on Debian 64bit then see if it works?

    yes

    Gotcha.

    Is this using the main source files from the website or using your git repo?

    I can also send you the built binaries. They have been compiled on Debian Testing (Bullseye), but I think they should work.

    Yep feel free to drop them in a filebox to 21:1/100 if able.

    I'll try and do those tests over the coming day.

    Just doing admin stuff at present. There's always admin to do. You'll love it they said, it'll be fun they said.. :)

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Mon Jan 25 09:05:09 2021
    Avon wrote (2021-01-25):

    10-4

    I had to look that up :)

    Is this using the main source files from the website or using your git repo?

    what is the URL of the website anyway? I don't remember where I downloaded soupgate.

    you need the fixes in the git repo.

    I can also send you the built binaries. They have been compiled on
    Debian Testing (Bullseye), but I think they should work.

    Yep feel free to drop them in a filebox to 21:1/100 if able.

    ok. I'll send a zip.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Mon Jan 25 21:25:40 2021
    On 25 Jan 2021 at 09:05a, Oli pondered and said...

    what is the URL of the website anyway? I don't remember where I
    downloaded soupgate.

    http://software.tomsweb.net/soupgate.html

    you need the fixes in the git repo.

    OK, I guess I should clone it again to be sure I get all the good stuff :)

    ok. I'll send a zip.

    Thanks

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Mon Jan 25 11:18:06 2021
    Avon wrote (2021-01-25):

    what is the URL of the website anyway? I don't remember where I
    downloaded soupgate.

    http://software.tomsweb.net/soupgate.html

    Good. The sgsources.zip is the same as the one in the gitlab repo. I wasn't sure that I had used the latest release.

    ok. I'll send a zip.

    done.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Tue Jan 26 20:45:30 2021
    On 25 Jan 2021 at 11:18a, Oli pondered and said...

    ok. I'll send a zip.

    done.

    It's been a bit of mare of an evening. The box I am using to setup this HUB would not post. So I had to phaff with the graphics card for a while.

    I then copied the zip contents over to it and none of them seemed to want to run.

    I'm just pulling down the repo again and will try compiling them on the box

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to Oli on Tue Jan 26 21:11:22 2021
    On 24 Jan 2021 at 04:14p, Oli pondered and said...

    apt install gcc-multilib g++-multilib

    g++ -m32 spoon.cpp config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp socket.cpp
    -o spoon
    gcc -m32 soupgate.c config.c lists.c pkt.c import.c export.c -o soupgate gcc -m32 pktview.c -o pktview

    all three compiled.

    spoon appeared to work puling down messages from NNTP.

    --- Spoon-Linux v1.06 Tue 26-Jan-2021 20:59:34
    Fetching news from xxx.xxx.xxx.xxx
    * Fetching alt.bbs.mystic
    There are 1 new messages (5896-5896) in this newsgroup.
    * Fetching comp.sys.cbm
    There are 3 new messages (3098-3100) in this newsgroup.
    * Fetching comp.sys.raspberry-pi
    There are 26 new messages (16674-16699) in this newsgroup.
    Average CPS fetching news: 143637
    * Fetching nz.test
    There are 5 new messages (200-204) in this newsgroup.
    Average CPS fetching news: 21168

    soupgate import failed and reported this error

    --- SoupGate-Linux v1.06 Tue 26-Jan-2021 21:01:02
    Importing from Soup to PKT
    Importing alt.bbs.mystic
    Error reading /hub/gateway/soup/00000000.msg
    Importing comp.sys.cbm
    Error reading /hub/gateway/soup/00000001.msg
    Importing comp.sys.raspberry-pi
    Error reading /hub/gateway/soup/00000002.msg
    Importing nz.test
    Error reading /hub/gateway/soup/00000003.msg
    No inbound messages found. Nothing to do.

    at the end of running import it appears to delete the AREAS file that s
  • From Avon@21:1/101 to Oli on Tue Jan 26 21:21:43 2021
    On 26 Jan 2021 at 09:11p, Avon pondered and said...

    I have not yet tested export for soupgate nor posting the exported files (assuming they work)

    I tested an export of a packet from HPT and this seemed to work

    Exporting from ?UT/?LO to Soup
    Exporting Type 2+ packet from 3:770/1 to 3:770/3
    1 messages exported
    mail: 0 news: 1

    This was an old packet I had posted to the NNTP server so it was interesting
    to test this also

    Spoon did try to post the packet, NNTP (rightfully) said No.

    Posting news to xxx.xxx.xxx.xxx
    * Posting article to nz.test
    NNTP host returned error: 441 435 Duplicate
    ! NNTP server did not accept message
    free(): double free detected in tcache 2
    Aborted


    Despite the failed post Spoon is not exiting correctly. This has been the
    same error I have seen in all my test compiles.

    free(): double free detected in tcache 2
    Aborted

    Not sure what that is or how to fix, but I can report when Soupgate creates
    an exported message it generates a number of .MSG files holding the gated
    posts to be sent to the NNTP and a REPLIES file that acts like an index of
    all the .MSG files to be sent.

    On a successful post session using Spoon the REPLIES should be deleted. But much like the error with Soupgate importing messages the REPLIES file
    remained undeleted.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to Oli on Tue Jan 26 21:37:54 2021
    On 26 Jan 2021 at 09:21p, Avon pondered and said...

    I tested an export of a packet from HPT and this seemed to work

    Exporting from ?UT/?LO to Soup
    Exporting Type 2+ packet from 3:770/1 to 3:770/3
    1 messages exported
    mail: 0 news: 1

    I re-ran this test again with a fresh packet to export, all seemed fine

    I then ran soup to post it and it did post, but again this error and the REPLIES file is left behind.


    Posting news to xxx.xxx.xxx.xxx
    * Posting article to nz.test
    free(): double free detected in tcache 2
    Aborted


    So it seems like there an issue with

    Soup - posting messages
    Soupgate - importing messages

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Tue Jan 26 09:43:56 2021
    Avon wrote (2021-01-26):

    spoon appeared to work puling down messages from NNTP.

    --- Spoon-Linux v1.06 Tue 26-Jan-2021 20:59:34
    Fetching news from xxx.xxx.xxx.xxx
    * Fetching alt.bbs.mystic
    There are 1 new messages (5896-5896) in this newsgroup.
    [...]
    soupgate import failed and reported this error

    --- SoupGate-Linux v1.06 Tue 26-Jan-2021 21:01:02
    Importing from Soup to PKT
    Importing alt.bbs.mystic
    Error reading /hub/gateway/soup/00000000.msg
    [...]
    No inbound messages found. Nothing to do.

    Could you send me (or put on hold) a zip with some Soup messages so I can try to import them and see if I find the error?

    at the end of running import it appears to delete the AREAS file that
    spoon created..

    It looks soupgate doesn't handle errors not always gracefully.

    I can also see three 0kb files created when I ran the import called

    00000000.TMP
    00000001.TMP
    00000002.TMP

    I wonder why it cant read the files it's trying to import.

    me too :)

    I have not yet tested export for soupgate nor posting the exported files (assuming they work)

    I successfully exported a PKT file to Soup, at least the Soup files looked fine (as far as I can tell). But my soupgate doesn't find anything, if I use the /pkt=pktdir parameter.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Tue Jan 26 21:53:12 2021
    On 26 Jan 2021 at 09:43a, Oli pondered and said...

    Importing from Soup to PKT
    Importing alt.bbs.mystic
    Error reading /hub/gateway/soup/00000000.msg
    [...]
    No inbound messages found. Nothing to do.

    Could you send me (or put on hold) a zip with some Soup messages so I
    can try to import them and see if I find the error?

    Will do. This will be tomorrow night (my time) as I'm just heading to bed now but sure thing :) I'll send you the .msg files and the other index file soup creates.

    at the end of running import it appears to delete the AREAS file that spoon created..

    It looks soupgate doesn't handle errors not always gracefully.

    Yep. I think it's correct to remove AREAS but ideally after it's used it to figure out which .msg files import to which area

    I can also see three 0kb files created when I ran the import called

    00000000.TMP
    00000001.TMP
    00000002.TMP

    I wonder why it cant read the files it's trying to import.

    me too :)

    These look to be working files created in the same directory I ran soupgate import from. There is a variable in the config for a temp directory

    TempDir

    But I suspect it was ignoring that at the time of import?

    I successfully exported a PKT file to Soup, at least the Soup files
    looked fine (as far as I can tell). But my soupgate doesn't find
    anything, if I use the /pkt=pktdir parameter.

    Make sure you export something from a tosser with a .bso file like a .clo
    that accompanies the PKT then running 'soupgate export' should work. Don't
    use .hlo files

    I too have had no joy with linux /pkt= switch, but it works fine for the windows version from the website. perhaps it's a pathing thing in the code or
    a case thing?

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Tue Jan 26 10:32:54 2021
    Avon wrote (2021-01-26):

    Posting news to xxx.xxx.xxx.xxx
    * Posting article to nz.test
    NNTP host returned error: 441 435 Duplicate
    ! NNTP server did not accept message
    free(): double free detected in tcache 2
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Aborted


    Despite the failed post Spoon is not exiting correctly. This has been the same error I have seen in all my test compiles.

    I think we need help from a Professional

    ---
    * Origin: . (21:3/102)
  • From tenser@21:1/101 to Avon on Wed Jan 27 02:59:47 2021
    On 26 Jan 2021 at 09:21p, Avon pondered and said...

    [snip]

    Spoon did try to post the packet, NNTP (rightfully) said No.

    Posting news to xxx.xxx.xxx.xxx
    * Posting article to nz.test
    NNTP host returned error: 441 435 Duplicate
    ! NNTP server did not accept message
    free(): double free detected in tcache 2
    Aborted

    Sorry, I haven't been keeping up here, but this is a bug
    in that software: the code is simply mismanaging memory.
    The solution here is to find the bug and fix it; sadly,
    much of this stuff is an unmaintained mess.... But trying
    a 32-bit build won't fix it, as was suggested earlier.

    Despite the failed post Spoon is not exiting correctly. This has been the same error I have seen in all my test compiles.

    free(): double free detected in tcache 2
    Aborted

    Not sure what that is or how to fix, but I can report when Soupgate creates an exported message it generates a number of .MSG files holding the gated posts to be sent to the NNTP and a REPLIES file that acts like an index of all the .MSG files to be sent.

    On a successful post session using Spoon the REPLIES should be deleted. But much like the error with Soupgate importing messages the REPLIES file remained undeleted.

    Well, a thing to try is to build it with one of the memory
    sanitizers turned on and see if you can find the location
    of the first free and the second; simply nil'ing out the pointer
    will likely move past _that_ problem, as free(NULL) is a no-op.
    But a more general problem is that none of this stuff was written
    to be memory safe; short of a substantial refactoring of the
    code, I suspect things like this will keep cropping up.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to tenser on Tue Jan 26 16:43:09 2021
    tenser wrote (2021-01-27):

    On 26 Jan 2021 at 09:21p, Avon pondered and said...
    free(): double free detected in tcache 2
    Aborted

    Sorry, I haven't been keeping up here, but this is a bug
    in that software: the code is simply mismanaging memory.
    The solution here is to find the bug and fix it; sadly,
    much of this stuff is an unmaintained mess.... But trying
    a 32-bit build won't fix it, as was suggested earlier.

    yes, these are separate issues, but building as 32-bit worked around some other issues.

    Well, a thing to try is to build it with one of the memory
    sanitizers turned on and see if you can find the location
    of the first free and the second; simply nil'ing out the pointer
    will likely move past _that_ problem, as free(NULL) is a no-op.

    I couldn't see anything obvious in the spoon code. I found some examples for building with the sanitizer turned on. Would something like this work:

    g++ spoon.cpp ... -o spoon -ggdb -fsanitize=address -fno-omit-frame-pointer

    or a longer version (not sure if the libs are needed):

    g++ spoon.cpp ... -o spoon -ggdb -fsanitize=address -fno-omit-frame-pointer -static-libstdc++ -static-libasan -lrt

    But a more general problem is that none of this stuff was written
    to be memory safe; short of a substantial refactoring of the
    code, I suspect things like this will keep cropping up.

    Rewrite everything in Go?

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Wed Jan 27 19:36:11 2021
    On 26 Jan 2021 at 09:43a, Oli pondered and said...

    Could you send me (or put on hold) a zip with some Soup messages so I
    can try to import them and see if I find the error?

    * Fetching alt.bbs.mystic
    There are 100 new messages (5797-5896) in this newsgroup.
    Average CPS fetching news: 49428
    * Fetching comp.sys.cbm
    There are 101 new messages (3001-3101) in this newsgroup.
    Average CPS fetching news: 162655
    * Fetching comp.sys.raspberry-pi
    There are 103 new messages (16600-16702) in this newsgroup.
    Average CPS fetching news: 249909

    I have created a file called oli-soup-imported-news.tar.gz

    Now how best to get it to you?

    * Origin: . (21:3/102)

    I'll netmail you with an idea.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to Oli on Wed Jan 27 19:54:57 2021
    On 26 Jan 2021 at 10:32a, Oli pondered and said...

    ! NNTP server did not accept message
    free(): double free detected in tcache 2
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Aborted

    Despite the failed post Spoon is not exiting correctly. This has been same error I have seen in all my test compiles.

    I think we need help from a Professional

    A Google search revealed this info.

    https://bit.ly/2YjfS5m

    trying to assign l from an uninitialised array str

    ..and this link seemed helpful.

    https://bit.ly/3cc3Ene

    [snip]

    Double free errors occur when free() is called more than once with the same memory address as an argument.

    Calling free() twice on the same value can lead to memory leak. When a
    program calls free() twice with the same argument, the program's memory management data structures become corrupted and could allow a malicious user
    to write values in arbitrary memory spaces. This corruption can cause the program to crash or, in some circumstances, alter the execution flow. By overwriting particular registers or memory spaces, an attacker can trick the program into executing code of their own choosing, often resulting in an interactive shell with elevated permissions.

    When a buffer is free()'d, a linked list of free buffers is read to rearrange and combine the chunks of free memory (to be able to allocate larger buffers
    in the future). These chunks are laid out in a double linked list which
    points to previous and next chunks. Unlinking an unused buffer (which is what happens when free() is called) could allow an attacker to write arbitrary values in memory; essentially overwriting valuable registers, calling
    shellcode from its own buffer.

    [snip]

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to tenser on Wed Jan 27 19:58:46 2021
    On 27 Jan 2021 at 02:59a, tenser pondered and said...

    Sorry, I haven't been keeping up here, but this is a bug
    in that software: the code is simply mismanaging memory.
    The solution here is to find the bug and fix it; sadly,
    much of this stuff is an unmaintained mess.... But trying
    a 32-bit build won't fix it, as was suggested earlier.

    I confess I'm mostly stumbling around in the dark here hoping something will work. My preference is to try to get the Linux code working if doable so as
    to be able to keep using what (until now) has been a relatively straight forward bit of software (for windows) and not have to figure out another more bulky option I have yet to find.

    I think the 32bit thing may have solved some of the issues... but problems remain with both soupgate import and soup export/posting to NNTP. :(

    On a successful post session using Spoon the REPLIES should be delete But much like the error with Soupgate importing messages the REPLIES remained undeleted.

    Well, a thing to try is to build it with one of the memory
    sanitizers turned on and see if you can find the location
    of the first free and the second; simply nil'ing out the pointer
    will likely move past _that_ problem, as free(NULL) is a no-op.
    But a more general problem is that none of this stuff was written
    to be memory safe; short of a substantial refactoring of the
    code, I suspect things like this will keep cropping up.

    Thank you for the guidance. I think Oli is perusing this and I'm trying to
    get him some news / soup files to play with.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Wed Jan 27 10:30:57 2021
    Avon wrote (2021-01-27):

    On 26 Jan 2021 at 09:43a, Oli pondered and said...

    Could you send me (or put on hold) a zip with some Soup messages so
    I can try to import them and see if I find the error?

    * Fetching alt.bbs.mystic
    There are 100 new messages (5797-5896) in this newsgroup.
    Average CPS fetching news: 49428
    * Fetching comp.sys.cbm
    There are 101 new messages (3001-3101) in this newsgroup.
    Average CPS fetching news: 162655
    * Fetching comp.sys.raspberry-pi
    There are 103 new messages (16600-16702) in this newsgroup.
    Average CPS fetching news: 249909

    I have created a file called oli-soup-imported-news.tar.gz

    SUCCESS!!!

    ../soupgate import
    SoupGate-Linux v1.06 - Fidonet/Internet gateway software
    Copyright (c) 1997-2000 by Tom Torfs, all rights reserved

    Beta version; compiled Jan 27 2021 10:21:11

    This is FREEWARE. Read soupgate.doc for more information.

    Importing from Soup to PKT

    Importing alt.bbs.mystic
    100 message(s) imported
    mail: 0 news: 100 lists: 0 junk: 0
    personal: 0 own org: 0 files: 0 ign/txt: 0
    Importing comp.sys.cbm
    101 message(s) imported
    mail: 0 news: 101 lists: 0 junk: 0
    personal: 0 own org: 0 files: 0 ign/txt: 0
    Importing comp.sys.raspberry-pi
    103 message(s) imported
    mail: 0 news: 103 lists: 0 junk: 0
    personal: 0 own org: 0 files: 0 ign/txt: 0

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Thu Jan 28 16:29:25 2021
    On 27 Jan 2021 at 10:30a, Oli pondered and said...

    SUCCESS!!!
    103 message(s) imported
    mail: 0 news: 103 lists: 0 junk: 0

    That's good stuff, if I pull down the latest version of the repo and compile it, do you think it would work for me too?

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Thu Jan 28 08:50:18 2021
    Avon wrote (2021-01-28):

    On 27 Jan 2021 at 10:30a, Oli pondered and said...

    SUCCESS!!!
    103 message(s) imported
    mail: 0 news: 103 lists: 0 junk: 0

    That's good stuff, if I pull down the latest version of the repo and compile it, do you think it would work for me too?

    Yes, I forgot to mention that.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Thu Jan 28 21:23:40 2021
    On 28 Jan 2021 at 08:50a, Oli pondered and said...

    That's good stuff, if I pull down the latest version of the repo and compile it, do you think it would work for me too?

    Yes, I forgot to mention that.

    Coolio, thanks Oli. I'll give it a try.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Thu Jan 28 10:39:29 2021
    Avon wrote (2021-01-28):

    On 28 Jan 2021 at 08:50a, Oli pondered and said...

    That's good stuff, if I pull down the latest version of the
    repo and compile it, do you think it would work for me too?

    Yes, I forgot to mention that.

    Coolio, thanks Oli. I'll give it a try.

    I just committed and pushed some 64-bit fixes. It might now work without the -m32 flag too.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to tenser on Fri Jan 29 21:21:35 2021
    On 27 Jan 2021 at 02:59a, tenser pondered and said...

    free(): double free detected in tcache 2
    Aborted

    Well, a thing to try is to build it with one of the memory
    sanitizers turned on and see if you can find the location
    of the first free and the second; simply nil'ing out the pointer
    will likely move past _that_ problem, as free(NULL) is a no-op.
    But a more general problem is that none of this stuff was written

    Could you guide me as to how to do this?

    I'm looking at the man for g++ but so far unsure how to enable a sanitizer mode.

    I'll keep poking and prodding :)

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to Oli on Fri Jan 29 23:09:41 2021
    On 28 Jan 2021 at 10:39a, Oli pondered and said...

    I just committed and pushed some 64-bit fixes. It might now work without the -m32 flag too.

    compiled the latest versions of soup and soupgate tonight

    I could pull down news and import it to PKT and could also export PKT back to news... only Soup News when posting is borking with the double free memory error after it successfully posts the gated news to the NNTP server.

    I spent some hours trying to learn what I might be able to do to debug

    Came across a tool called Clang that has a -fsanitize=address option

    I'm not really sure what I'm doing with this one :)


    clang -fsanitize=address -O1 -fno-omit-frame-pointer -g spoon.cpp config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp socket.cpp
    pop.cpp:294:71: warning: format specifies type 'int' but the argument has
    type 'long' [-Wformat]
    printf("Examining headers of message %d of %d...\n", i+1, nummessages);
    ~~ ^~~~~~~~~~~
    %ld
    pop.cpp:382:22: warning: format specifies type 'int' but the argument has
    type 'long' [-Wformat]
    i+1, nummessages, msglist[i].msgsize);
    ^~~~~~~~~~~
    2 warnings generated.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Fri Jan 29 20:43:46 2021
    Avon wrote (2021-01-29):

    Came across a tool called Clang that has a -fsanitize=address option

    That's a C compiler.

    I'm not really sure what I'm doing with this one :)


    clang -fsanitize=address -O1 -fno-omit-frame-pointer -g spoon.cpp config.cpp keyboard.cpp nntp.cpp pop.cpp smtp.cpp socket.cpp pop.cpp:294:71: warning: format specifies type 'int' but the argument has type 'long' [-Wformat]
    printf("Examining headers of message %d of %d...\n", i+1, nummessages);
    ~~
    ^~~~~~~~~~~
    %ld
    pop.cpp:382:22: warning: format specifies type 'int' but the argument has type 'long' [-Wformat]
    i+1, nummessages, msglist[i].msgsize);
    ^~~~~~~~~~~
    2 warnings generated.

    Compile time warnings. Easy to fix, but will not help with the NNTP bug. pop.cpp is for POP3 mail fetch.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Sat Jan 30 10:05:28 2021
    On 29 Jan 2021 at 08:43p, Oli pondered and said...

    Came across a tool called Clang that has a -fsanitize=address option

    That's a C compiler.

    Oh OK, yeah I was trying to find a tool to follow up on Tensers suggestion of looking for the double free memory issue.

    I can understand how it can be a pain to try and track down as (if I
    understand this right) we're essentially looking for locations across the
    code where a variable is being called again but has already been
    closed/nulled? It seems like there are tools about that can check the code on compile and alert to hints as to where/what may be going on?

    2 warnings generated.

    Compile time warnings. Easy to fix, but will not help with the NNTP bug. pop.cpp is for POP3 mail fetch.

    Yep that's how I interpreted it also... interesting to see the issues in the pop code, not that I use it... I've been trying to understand the nntp code
    and the area around when a post is made and then the error occurs.

    I'm a 2 year old trying to read a scientific novel :)

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Sat Jan 30 08:53:21 2021
    Avon wrote (2021-01-30):

    On 29 Jan 2021 at 08:43p, Oli pondered and said...

    Came across a tool called Clang that has a -fsanitize=address
    option

    That's a C compiler.

    Oh OK, yeah I was trying to find a tool to follow up on Tensers
    suggestion of looking for the double free memory issue.

    (of course Clang is also a C++ and Objective C compiler)

    I can understand how it can be a pain to try and track down as (if I understand this right) we're essentially looking for locations across the code where a variable is being called again but has already been closed/nulled? It seems like there are tools about that can check the
    code on compile and alert to hints as to where/what may be going on?

    The compiler is the right tool for this. If I understand it correctly you would get the warnings / hints at runtime. Did you try to run the clang compiled spoon

    Compile time warnings. Easy to fix, but will not help with the NNTP
    bug. pop.cpp is for POP3 mail fetch.

    Yep that's how I interpreted it also... interesting to see the issues in the pop code, not that I use it... I've been trying to understand the
    nntp code and the area around when a post is made and then the error occurs.

    I think it is in the socket.cpp code. There are a couple of deletes.

    ---
    * Origin: . (21:3/102)
  • From Oli@21:3/102 to Avon on Sat Jan 30 11:42:06 2021
    Avon wrote (2021-01-29):

    2 warnings generated.

    fixed. see the changes with

    git diff a3eac^ a3eac

    ---
    * Origin: . (21:3/102)
  • From Rushfan@21:2/115 to Oli on Sat Jan 30 21:43:49 2021
    BY: Oli(21:3/102)


    |11O|09> |10Avon wrote (2021-01-29):|07
    |11O|09> |07
    |11O|09> |10A> 2 warnings generated.|07
    |11O|09> |07
    |11O|09> |10fixed. see the changes with|07
    |11O|09> |07
    |11O|09> |10git diff a3eac^ a3eac|07

    What's the git repo for these changes (I tried searching through the messages but didn't find it).

    Also which pktview are you using? I'll try to fix that for 64-bit. |11r|09ushfan|07


    --- WWIV 5.6.0.3400
    * Origin: Mystic Rhythms BBS (21:2/115)
  • From Oli@21:3/102 to Rushfan on Sun Jan 31 07:46:02 2021
    Rushfan wrote (2021-01-30):

    BY: Oli(21:3/102)


    |11O|09>> |10Avon wrote (2021-01-29):|07
    |11O|09>> |07
    |10A>>> 2 warnings generated.|07
    |11O|09>> |07
    |11O|09>> |10fixed. see the changes with|07
    |11O|09>> |07
    |11O|09>> |10git diff a3eac^ a3eac|07

    interesting quoting signs

    What's the git repo for these changes (I tried searching through the messages but didn't find it).

    https://gitlab.com/fidosoft/soupgate

    Also which pktview are you using? I'll try to fix that for 64-bit.

    Should be all fixed for 64-bit, same bug in soupgate and pktview. It was just one long in the packet header structure that must be 32-bit, but is 64-bit long on Linux.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Rushfan on Sun Jan 31 19:57:33 2021
    On 30 Jan 2021 at 09:43p, Rushfan pondered and said...

    What's the git repo for these changes (I tried searching through the messages but didn't find it).

    If you are able to look at Soup and deduce the double free issue when
    posting to NNTP that would be appreciated.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From tenser@21:1/101 to Avon on Mon Feb 1 06:44:18 2021
    On 29 Jan 2021 at 09:21p, Avon pondered and said...

    On 27 Jan 2021 at 02:59a, tenser pondered and said...

    free(): double free detected in tcache 2
    Aborted

    Well, a thing to try is to build it with one of the memory
    sanitizers turned on and see if you can find the location
    of the first free and the second; simply nil'ing out the pointer
    will likely move past _that_ problem, as free(NULL) is a no-op.
    But a more general problem is that none of this stuff was written

    Could you guide me as to how to do this?

    I'm looking at the man for g++ but so far unsure how to enable a
    sanitizer mode.

    I'll keep poking and prodding :)

    Sure. First, perhaps it makes some sense to briefly describe
    how we go from source code to an executable image in a Unix-style
    environment, specifically using C/C++ compilers.

    For most medium- to large-size programs, the basic gist of it is
    that there are two logically separate steps:

    1. Compilation. This is where we invoke the compiler on a
    source file (e.g., .c/.cc/.cpp/whatever). The compiler
    will read in the source file and, if it's a valid
    compilable file, will emit an "object file": this is the
    ".o" file we see after running the compiler. Usually
    the object file is in the "ELF" format.
    2. Linking. This is where we invoke the linker on a set of
    object files and library archives to produce an executable
    image. The linker is responsible for combining object
    files and resolving references between then and libraries.

    Consider a simple program such as "Hello World": in C++, we
    would write:

    #include <iostream>

    int
    main()
    {
    std::cout << "Hello, World!" << std::endl;
    return 0;
    }

    Suppose we put that text into "hello.cc". Then we can compile
    it by running two commands:

    % c++ -c hello.cc
    % c++ -o hello hello.o

    The first creates an object file called "hello.o". The second
    invokes the linker to create an executable file called "hello"
    that's created from whatever libraries the "c++" _driver command_
    claims are necessary for C++ and from the "hello.o" object file.
    For simple programs like this, we can invoke the compiler driver
    program in such a way that the two steps are combined:

    % c++ -o hello hello.cc

    But note that these two independent steps still happen; I've used
    the term "driver command" for the `c++` program, because that's
    what it is: it's a program that actually invokes the real programs
    that make up the compiler and linker.

    Okay, why is any of this relevant? Because the sanitizers need
    support from both the compiler _and_ the linker. You tell both
    to enable sanitizer support by specifying command line flags to
    the driver program: to enable address sanitizer (which will help
    you find use-after-free and double-free bugs) you want to give
    the `-fsanitize=address` argument. So I could try to compile
    e.g. 'soup' as:

    % c++ -g -fsanitize=address -o soup foo.cpp bar.cpp etc.cpp

    In other words, edit the `makesoup` shell script and add
    `-g -fsanitize=address` to the `g++` or `c++` line.

    Then, when you run the resulting binary, it should tell you
    where the double-free occurs.

    This works since the `makesoup` shell script invokes the driver
    program so that it compiles _and_ links the `soup` source files
    in a single invocation.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to tenser on Mon Feb 1 11:56:01 2021
    On 01 Feb 2021 at 06:44a, tenser pondered and said...

    In other words, edit the `makesoup` shell script and add
    `-g -fsanitize=address` to the `g++` or `c++` line.

    Then, when you run the resulting binary, it should tell you
    where the double-free occurs.

    Thanks :) I'll try to do this over the coming days. A bit time poor at
    present and need to also add some new nodes to the network etc.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)