It seems a lot of messages were being rescanned for no reason. Has that been an issue for others recently?


--- WWIV 5.7.0.3455
* Origin: inland utopia * socal usa * iutopia.duckdns.org:2023 (21:4/108)

Re: rescan
By: Utopian Galt to All on Sat Mar 13 2021 08:34 pm

It seems a lot of messages were being rescanned for no reason. Has that been an issue for others recently?

It can happen from time to time. Maybe a node did a rescan or???

I just got a bunch of old messages come through but they all went to the bad area here.

Ttyl :-),
Al

... No amount of planning will replace dumb luck
--- SBBSecho 3.13-Linux
* Origin: The Rusty MailBox - Penticton, BC Canada (21:4/106.1)

Hi Utopian,

It seems a lot of messages were being rescanned for no reason. Has
that been an issue for others recently?

Hmm can you see where they came from?

At 1/100 I can see the HUB picked up a handfull from 2/100 that seem to have failed a date check i.e. too old.. Not sure why they were sent out.

I do know Solaris was doing some work on a NET 2 node he runs, perhaps a rescan went rogue (purely a guess).?

Best, Paul.

--- GoldED+/LNX 1.1.5-b20180707
* Origin: Agency + Risa HUB - Dunedin, New Zealand (21:1/100)

Avon wrote (2021-03-14):

Hi Utopian,

It seems a lot of messages were being rescanned for no reason. Has
that been an issue for others recently?

Hmm can you see where they came from?

At 1/100 I can see the HUB picked up a handfull from 2/100 that seem to have failed a date check i.e. too old.. Not sure why they were sent out.

I do know Solaris was doing some work on a NET 2 node he runs, perhaps a rescan went rogue (purely a guess).?

Best, Paul.

Your system successfully demonstrated the 2-second/wrong-time-on-rescan bug of hpt (SMAPI) for real.

---
* Origin: . (21:3/102)

Am 13.03.21 schrieb Utopian Galt@21:4/108 in FSX_NET:

Hallo Utopian,

It seems a lot of messages were being rescanned for no reason. Has that been an issue for others recently?

I also received a lot of old messages in different FSX_* echos today...
And I did not change my fsxNet setup on my node (Synchronet) today :)

Regards,
Anna

--- OpenXP 5.0.49
* Origin: Imzadi Box Point (21:3/127.1)

Re: rescan
By: Oli to Avon on Sun Mar 14 2021 10:11 am

Your system successfully demonstrated the 2-second/wrong-time-on-rescan bug of hpt (SMAPI) for real.

I'm not following...?

I'm wondering how a node doing a rescan can inject dupes into the network? Unless it was a "hub" that did the rescan from another node?

(A quick check revealed my dups came via 2/1202 -> 2/100, but the origin of the messages came from varios hub 1 nodes.)

But the 2 second bug in hpt wouldnt trap those? (I'm running an oldish build of hpt (probably due for an update as I have seen some development activity), and hub 3 tossed them all into the dupe message base (from what I can tell) - I'm assuming because of the msgid.)

...ëîåï

... Never drink black coffee at lunch. It will keep you awake in the afternoo --- SBBSecho 3.12-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

deon wrote (2021-03-14):

Re: rescan
By: Oli to Avon on Sun Mar 14 2021 10:11 am

Your system successfully demonstrated the
2-second/wrong-time-on-rescan bug of hpt (SMAPI) for real.

I'm not following...?

I'm wondering how a node doing a rescan can inject dupes into the
network? Unless it was a "hub" that did the rescan from another node?

(A quick check revealed my dups came via 2/1202 -> 2/100, but the origin
of the messages came from varios hub 1 nodes.)

But the 2 second bug in hpt wouldnt trap those? (I'm running an oldish build of hpt (probably due for an update as I have seen some development activity), and hub 3 tossed them all into the dupe message base (from
what I can tell) - I'm assuming because of the msgid.)

I don't know. I received a couple of hundred dupes. I guess messages before Nov/Dec 2020 weren't in hub 3s dupe database.

My point is that this is a real world example that network-wide dupes caused by a rescan do happen. And that hpt does create unnecessary problems.

All dupes I see in the message base have an odd number of seconds in the original message's time.

q.e.d.

---
* Origin: . (21:3/102)

On 14 Mar 2021 at 11:08p, deon pondered and said...

(A quick check revealed my dups came via 2/1202 -> 2/100, but the origin of the messages came from varios hub 1 nodes.)

I did offer 2/1202 a feed from 1/100 so guessing that a rescan to him from 1/100 was tossed to 2/100 and onwards. I think...

Your system successfully demonstrated the 2-second/wrong-time-on-resc bug of hpt (SMAPI) for real.

Have no idea about that. But I am running the latest HPT where as 2/100 is still on a dated Mystic (for now).

--- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

Re: Re: rescan
By: Avon to deon on Mon Mar 15 2021 10:02 am

(A quick check revealed my dups came via 2/1202 -> 2/100, but the origin of the messages came from varios hub 1 nodes.)

I did offer 2/1202 a feed from 1/100 so guessing that a rescan to him from 1/100 was tossed to 2/100 and onwards. I think...

Ahh, that would do it.

In a perfect world, 2/100 should have seen them as dupes then and not send them upstream.

Need to work on Mr Solaris and get him to switch to hpt as a hub too - the plus being 1 less "BBS" to manage :)

...ëîåï

... When you dial a wrong number, you NEVER get a busy signal.
--- SBBSecho 3.12-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

Re: rescan
By: Oli to deon on Sun Mar 14 2021 03:01 pm

I don't know. I received a couple of hundred dupes. I guess messages before Nov/Dec 2020 weren't in hub 3s dupe database.

Thinking about it, since Hub 3 put them (the ones I saw anyway) in the dupe base, you shouldnt have got them.

Can you give me an example - so I can try and see why they were passed on... (Anna, perhaps you could too.) It would be helpful to have the date/packet you received and a couple of msgids (and date of messages) so that I can hunt them down in the logs.

I feed my BBS off of Hub 3 (as well as Hub 2) - and I didnt see them.

...ëîåï

... When you smell an odourless gas, it is probably carbon monoxide.
--- SBBSecho 3.12-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

Avon wrote (2021-03-15):

Your system successfully demonstrated the
2-second/wrong-time-on-resc bug of hpt (SMAPI) for real.

Have no idea about that. But I am running the latest HPT where as 2/100 is still on a dated Mystic (for now).

Mystic would have done the rescan/export correctly (I think), but HPT does change every odd number of seconds to even seconds. 08:24:01 gets exported as 08:24:00. This has been recently fixed for Squish bases, but not for JAM.

---
* Origin: . (21:3/102)

deon wrote (2021-03-15):

Re: rescan
By: Oli to deon on Sun Mar 14 2021 03:01 pm

I don't know. I received a couple of hundred dupes. I guess
messages before Nov/Dec 2020 weren't in hub 3s dupe database.

Thinking about it, since Hub 3 put them (the ones I saw anyway) in the
dupe base, you shouldnt have got them.

Can you give me an example - so I can try and see why they were passed on... (Anna, perhaps you could too.) It would be helpful to have the date/packet you received and a couple of msgids (and date of messages) so that I can hunt them down in the logs.

I feed my BBS off of Hub 3 (as well as Hub 2) - and I didnt see them.

Interesting. This is a packet I received that contains only dupes:

1706838 Mar 14 03:44 4d86ef01.pkt
(time in UTC)

I'll return it to sender ... ;)

---
* Origin: . (21:3/102)

Re: rescan
By: Oli to deon on Mon Mar 15 2021 08:51 am

Can you give me an example - so I can try and see why they were passed on... (Anna, perhaps you could too.) It would be
helpful to have the date/packet you received and a couple of msgids (and date of messages) so that I can hunt them down in the
logs.

Interesting. This is a packet I received that contains only dupes:
1706838 Mar 14 03:44 4d86ef01.pkt

OK, Synchronet is such a better BBS package (when handling mail at least) :)

Found them:

7 14:44:15 pkt: /fido/mailer/in.tmp/083c6201.tos [21:2/100]
4 14:44:19 Areas summary:
4 14:44:19 dupe area dupe - 447 msgs
4 14:44:19 echo area FSX_MYS - 254 msgs
4 14:44:19 echo area FSX_BBS - 224 msgs
4 14:44:19 echo area FSX_ENG - 96 msgs
4 14:44:19 echo area FSX_CRY - 257 msgs
4 14:44:19 echo area FSX_NET - 232 msgs

Hub 3 saved you from getting only 447 of them - the rest would have been passed on - probably because they were old messages and Hub 3 only has 3 months of dupe history (I'm going to increase that to 3yrs - but it'll take some time to get there.)

I'm not aware of a config item to limit the age of received messages - if you are let me know and I'll implement that too.

I (2/116) never got the messages from Hub 3 - since they all dig originate from 2/100 - and from Hub 3 2/116 would have been in the seen-by's already - so Hub 3 never packaged them up for me.

That said, I did get them to 2/116 from 2/100 - and SBBS discarded them because of age and dupe checking. :)

...ëîåï

... Distrust all in whom the impulse to punish is powerful.
--- SBBSecho 3.12-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

I'm not aware of a config item to limit the age of received messages - if you are let me know and I'll implement that too.

SBBSecho and hpt both have options to put aside messages older than a certain time. I use -tooOld 60 with hpt and MaxEchoMailAge = 30D in sbbsecho.ini.

I'm not sure what BBBS would do with those old messages. They would get trapped as dupes if they were in the dupe database I guess.

--- BBBS/Li6 v4.10 Toy-4
* Origin: The Rusty MailBox - Penticton, BC Canada (21:4/106.2)

Al wrote (2021-03-15):

I'm not aware of a config item to limit the age of received messages -
if you are let me know and I'll implement that too.

SBBSecho and hpt both have options to put aside messages older than a certain time. I use -tooOld 60 with hpt and MaxEchoMailAge = 30D in sbbsecho.ini.

I'm not sure what BBBS would do with those old messages. They would get trapped as dupes if they were in the dupe database I guess.

Does any tosser have the capability to check against messages stored in the message base or populate the dupe database from messages in the message base?

---
* Origin: . (21:3/102)

deon wrote (2021-03-15):

Re: rescan
By: Oli to deon on Mon Mar 15 2021 08:51 am

Can you give me an example - so I can try and see why they were
passed on... (Anna, perhaps you could too.) It would be helpful to
have the date/packet you received and a couple of msgids (and date
of messages) so that I can hunt them down in the logs.

Interesting. This is a packet I received that contains only dupes:
1706838 Mar 14 03:44 4d86ef01.pkt

OK, Synchronet is such a better BBS package (when handling mail at least) :)

Found them:

7 14:44:15 pkt: /fido/mailer/in.tmp/083c6201.tos [21:2/100]
4 14:44:19 Areas summary:
4 14:44:19 dupe area dupe - 447 msgs
4 14:44:19 echo area FSX_MYS - 254 msgs
4 14:44:19 echo area FSX_BBS - 224 msgs
4 14:44:19 echo area FSX_ENG - 96 msgs
4 14:44:19 echo area FSX_CRY - 257 msgs
4 14:44:19 echo area FSX_NET - 232 msgs

Hub 3 saved you from getting only 447 of them - the rest would have been passed on - probably because they were old messages and Hub 3 only has 3 months of dupe history (I'm going to increase that to 3yrs - but it'll
take some time to get there.)

My crashmail detected a couple of newer mails that went through as dupes (Nov/Dec, dupe database 3 month + some weeks).

That said, I did get them to 2/116 from 2/100 - and SBBS discarded them because of age and dupe checking. :)

It seems Mystic does not like the modified time of the rescanned messages when it comes to dupe checking. Or is there another explanation why I only received odd-second dupes?

---
* Origin: . (21:3/102)

Al wrote (2021-03-15):

rom: "Al -> deon" <2@106.4.21>
Subject: rescan
Date: Mon, 15 Mar 2021 03:19:14 -0800
Newsgroups: FSX_NET

Your time or timezone is wrong.

---
* Origin: . (21:3/102)

Re: rescan
By: Oli to Al on Mon Mar 15 2021 11:32 am

Does any tosser have the capability to check against messages stored in the message base or populate the dupe database from messages in the message base?

They all have their own way to do it. In Synchronet I keep 500 messages in each dupe base. In hpt I keep a dupe history of 90 and that's enough since I reject echomail over 60 days.

BBBS keeps 20x1024 in a single dupebase for all areas. I'm note sure how long it takes to go over that number but I can increase or decrease the number of dupes kept.

All those packages do a good job of detecting dupes.

Ttyl :-),
Al

... Discoveries are often made by not following instructions.
--- SBBSecho 3.13-Linux
* Origin: The Rusty MailBox - Penticton, BC Canada (21:4/106.1)

Al wrote (2021-03-15):

rom: "Al -> deon" <2@106.4.21>
Subject: rescan
Date: Mon, 15 Mar 2021 03:19:14 -0800
Newsgroups: FSX_NET

Your time or timezone is wrong.

Yes, BBBS does a bad job of time zones. I just reset the TZUTC and TZ environments for BBBS and it still missed it for some reason.

BBBS will stumble over that once in a while, I don't know why. Outgoing tic files seem to be stamped -0800 all year long, even in PST like now when it should be -0700.

--- BBBS/Li6 v4.10 Toy-4
* Origin: The Rusty MailBox - Penticton, BC Canada (21:4/106.2)

Re: rescan
By: Al to deon on Mon Mar 15 2021 03:19 am

Thanks Al,

SBBSecho and hpt both have options to put aside messages older than a certain time. I use -tooOld 60 with hpt and MaxEchoMailAge =
30D in sbbsecho.ini.

I thought I'd see a config item -tooOld.

OK, Hub 3 is configured with -tooOld 90, so with it and the dupe checking, that should stop any stray batch of old message entering again.

...ëîåï

... Ahhhhhhhh, I forget what I was going to say.
--- SBBSecho 3.12-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

Re: rescan
By: Oli to deon on Mon Mar 15 2021 11:46 am

It seems Mystic does not like the modified time of the rescanned messages when it comes to dupe checking. Or is there another
explanation why I only received odd-second dupes?

Hmm... Good question.

So lets recap, so we are on the same page.

* 2/1202 rescanned from 1/100 - I'm assuming it got 365 days worth of some messages (based on the age that my BBS discarded them). Although the numbers are low for a years worth, so some were stopped before they got to 2/116 and 3/100.

* 2/1202 sent (some of) them to 2/100

* 2/116 received them (from 2/100) and discarded them:
2021-03-14 14:44:30 Duplicate: 84 detected in FSX_GEN
2021-03-14 14:44:30 Duplicate: 44 detected in FSX_MYS
2021-03-14 14:44:30 Duplicate: 28 detected in FSX_BBS
2021-03-14 14:44:30 Duplicate: 4 detected in FSX_ENG
2021-03-14 14:44:30 Duplicate: 78 detected in FSX_BOT
2021-03-14 14:44:30 Duplicate: 198 detected in FSX_DAT
2021-03-14 14:44:30 Duplicate: 54 detected in FSX_NET

490 messages. (none were imported).
1456 discarded due to age. (total 1946).

* 3/100 received them (from 2/100), threw some into dupes and sent the rest on 4 14:44:19 Areas summary:
4 14:44:19 dupe area dupe - 447 msgs
4 14:44:19 echo area FSX_MYS - 254 msgs
4 14:44:19 echo area FSX_BBS - 224 msgs
4 14:44:19 echo area FSX_ENG - 96 msgs
4 14:44:19 echo area FSX_CRY - 257 msgs
4 14:44:19 echo area FSX_NET - 232 msgs

1510 messages. 1063 passed on.

So based on what I know about the hpt jam/squish bug - rescans change the time - to an odd time? Hence why you got odd messages (since they all originated from 1/100).

Dont know what happened to the other 436 that my BBS got but Hub 3 didnt report?

...ëîåï

... When a man brings his wife flowers for no reason - there's a reason.
--- SBBSecho 3.12-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

Am 15.03.21 schrieb deon@21:2/116 in FSX_NET:

Hallo deon,

Can you give me an example - so I can try and see why they were passed
on... (Anna, perhaps you could too.)

I guess this packet contained a lot of old messages:

2021-03-14 04:45:07 Importing /sbbs/fido/inbound/4d86ef14.pkt
(Type 2e, 1666.8KB) from 21:3/100 to 21:3/127

It is unusual for fsxNet packets to be 1.6 MB in size :)

An example of SBBSecho log entries:

2021-03-14 04:45:07 ERROR 1 (smb_addmsg duplicate X-FTN-MSGID: 21:1/158
bb44b6cf found in message #3) line 3617 adding message to fsxnet_fsx_mys 2021-03-14 04:45:07 FSX_MYS Duplicate message from MeaTLoTioN (21:1/158)
to Andy, subject: Re: Reading Netmail - Pressing ESC
2021-03-14 04:45:07 ERROR 1 (smb_addmsg duplicate X-FTN-MSGID: 21:1/158.6
9738e10c found in message #13) line 3617 adding message to fsxnet_fsx_mys 2021-03-14 04:45:07 FSX_MYS Duplicate message from MeaTLoTioN (21:1/158.6)
to Charles Pierson, subject: Re: I feel like an idiot
2021-03-14 04:45:07 FSX_MYS: Failed to parse Origin Line in message from
Barmed (21:3/100) in packet from 21:3/100: /sbbs/fido/inbound/
4d86ef14.pkt
2021-03-14 04:45:07 ERROR 1 (smb_addmsg duplicate X-FTN-MSGID: 21:1/158.6
77e7c572 found in message #20) line 3617 adding message to fsxnet_fsx_mys 2021-03-14 04:45:07 FSX_MYS Duplicate message from Barmed (21:3/100) to
paulie420, subject: Re: I feel like an idiot

The summary is:

2021-03-14 04:45:07 Imported: 251 msgs fsxnet_fsx_mys <- FSX_MYS
2021-03-14 04:45:07 Imported: 212 msgs fsxnet_fsx_bbs <- FSX_BBS
2021-03-14 04:45:07 Imported: 90 msgs fsxnet_fsx_eng <- FSX_ENG
2021-03-14 04:45:07 Imported: 257 msgs fsxnet_fsx_cry <- FSX_CRY
2021-03-14 04:45:07 Imported: 229 msgs fsxnet_fsx_net <- FSX_NET
2021-03-14 04:45:07 Duplicate: 3 detected in FSX_MYS
2021-03-14 04:45:07 Duplicate: 12 detected in FSX_BBS
2021-03-14 04:45:07 Duplicate: 6 detected in FSX_ENG
2021-03-14 04:45:07 Duplicate: 3 detected in FSX_NET
2021-03-14 04:45:07 Imported: 1039 msgs total

Regards,
Anna



--- OpenXP 5.0.49
* Origin: Imzadi Box Point (21:3/127.1)

On 15 Mar 2021 at 10:34p, deon pondered and said...

OK, Hub 3 is configured with -tooOld 90, so with it and the dupe
checking, that

I'd set HUB 1 to 60 but am wondering if it should be even lower like 50?

Ideally we all set ours the same too..

I'm hard pressed to think of messages older than six weeks that should be allowed to pass unchecked.

In time we'll work on HUB 2 to bring into line as well. I'm focused on learning/configuting HUB 1 first so I can help Todd when the time comes.

--- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

Re: Re: rescan
By: Avon to deon on Tue Mar 16 2021 11:25 am

OK, Hub 3 is configured with -tooOld 90, so with it and the dupe
checking, that

I'd set HUB 1 to 60 but am wondering if it should be even lower like 50?

I'm OK with 50 or 30 or something.

I cant think of a valid scenario to accept a bunch of old messages into the network (but nodes can still rescan for more than that to be sent to them).

With de-dupe set at at least that as well, should catch any accidental burps.

In time we'll work on HUB 2 to bring into line as well. I'm focused on learning/configuting HUB 1 first so I can help Todd when the time comes.

So, I would suggest you and Todd consider going my docker route. Dan and I are already there.

Everything you do is the same - managing config files, the thing that docker provides is the application and all its dependancies are ready to go - and all in a consistent location. (You can manage, view, edit those config files and logs outside of the container as well.)

Outside of the container (ie: the host), you can still put stuff where you want it /avon if you wanted to - but inside, binkd is /etc/binkd, hpt is /etc/ftn etc. I have all "hub data" in /fido (inside the container, ie: inbound, outbound, msgbases, etc)

I have my "tools" (ie: the bot) in /usr/local/tools (inside the container - completely different path outside it) - so my config files reference /usr/local/tools/filter... etc.

But which ever way you go, I think it would be a plus that Todd changes the Mystic Hub, to hopefully avoid any other glitches like this.

...ëîåï

... Components that must not and cannot be assembled improperly will be.
--- SBBSecho 3.12-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

On 16 Mar 2021 at 11:56a, deon pondered and said...

I'm OK with 50 or 30 or something.

Coolio... we'll discuss as a group and set something up.

In time we'll work on HUB 2 to bring into line as well. I'm focused o learning/configuting HUB 1 first so I can help Todd when the time com

So, I would suggest you and Todd consider going my docker route. Dan and
I are already there.

In time I'm sure we will but for now I am more than happy to learn Linux by doing it myself albeit slowly and painfully for me at times. But I want to learn by doing vs having it all turnkey.

In Todds case I was thinking of setting up the Windows versions with him but
we need to have a chat about what his prefs are for 2/100 and go from there. Right now I am still working on getting things working as I want for 1/100
and still to migrate over a bunch of things from my old windows systems to
the new box which I want to focus on first.

But which ever way you go, I think it would be a plus that Todd changes the Mystic Hub, to hopefully avoid any other glitches like this.

Agreed but I don't see it as pressing.

--- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

deon wrote (2021-03-16):

But which ever way you go, I think it would be a plus that Todd changes
the Mystic Hub, to hopefully avoid any other glitches like this.

Isn't this kind of victim blaming? ;) I believe Mystic would have catched all dupes, if hpt hadn't modified the time of the rescanned mails.

---
* Origin: . (21:3/102)

deon wrote (2021-03-15):

Re: rescan
By: Oli to deon on Mon Mar 15 2021 11:46 am

It seems Mystic does not like the modified time of the rescanned
messages when it comes to dupe checking. Or is there another
explanation why I only received odd-second dupes?

Hmm... Good question.

So lets recap, so we are on the same page.

* 2/1202 rescanned from 1/100 - I'm assuming it got 365 days worth of
some messages (based on the age that my BBS discarded them). Although the numbers are low for a years worth, so some were stopped before they got
to 2/116 and 3/100.

* 2/1202 sent (some of) them to 2/100

Or all of them. My theory: 2/100 detected all messages with the correct and original time as dupes and forwarded the ones with the modified time. Which means all messages that had originally odd seconds in the datetime header field went through.

Do we know how Mystic's dupe detection work?

* 2/116 received them (from 2/100) and discarded them:
2021-03-14 14:44:30 Duplicate: 84 detected in FSX_GEN
2021-03-14 14:44:30 Duplicate: 44 detected in FSX_MYS
2021-03-14 14:44:30 Duplicate: 28 detected in FSX_BBS
2021-03-14 14:44:30 Duplicate: 4 detected in FSX_ENG
2021-03-14 14:44:30 Duplicate: 78 detected in FSX_BOT
2021-03-14 14:44:30 Duplicate: 198 detected in FSX_DAT
2021-03-14 14:44:30 Duplicate: 54 detected in FSX_NET

490 messages. (none were imported).
1456 discarded due to age. (total 1946).

* 3/100 received them (from 2/100), threw some into dupes and sent the
rest on 4 14:44:19 Areas summary:
4 14:44:19 dupe area dupe - 447 msgs
4 14:44:19 echo area FSX_MYS - 254 msgs
4 14:44:19 echo area FSX_BBS - 224 msgs
4 14:44:19 echo area FSX_ENG - 96 msgs
4 14:44:19 echo area FSX_CRY - 257 msgs
4 14:44:19 echo area FSX_NET - 232 msgs

1510 messages. 1063 passed on.

So based on what I know about the hpt jam/squish bug - rescans change the time - to an odd time? Hence why you got odd messages (since they all originated from 1/100).

It's processed internally as DOS time, which only has 2-second resolution. This is equivalent to set the least significant bit of a unix time to 0. The dupes I received had an even number of seconds, but when I compare them to the ones I originally received and have still in my message base, I see odd seconds.

modified message received as dupe: 20:05:22
in message base received last year: 20:05:23

I haven't checked every single dupe message, but I couldn't find any message with even seconds in my message base (from the ones I received as dupe). But I might be missing something.

---
* Origin: . (21:3/102)

Re: rescan
By: Oli to deon on Tue Mar 16 2021 09:45 am

Isn't this kind of victim blaming? ;) I believe Mystic would have catched all dupes, if hpt hadn't modified the time of the rescanned
mails.

But it didnt though? Or did I miss something?

Are you suggesting that the date and MSGID are both used for dupe checking by Mystic, and thus because the messages had a different date (but same MSGID) that's why Mystic let them through?

...ëîåï

... A gentleman is one who never swears at his wife while ladies are present. --- SBBSecho 3.12-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

Re: rescan
By: Oli to deon on Tue Mar 16 2021 10:07 am

Do we know how Mystic's dupe detection work?

Not very well? (again) :P

I know some BBSes use different (additional) methods for dupe detection, but I thought that the "minimum" should have been checking that the MSGID hasnt been used within the last 3 years.

...ëîåï

... I have a new philosophy. I'm only going to dread one day at a time.
--- SBBSecho 3.12-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

deon wrote (2021-03-16):

Re: rescan
By: Oli to deon on Tue Mar 16 2021 09:45 am

Isn't this kind of victim blaming? ;) I believe Mystic would have
catched all dupes, if hpt hadn't modified the time of the rescanned
mails.

But it didnt though? Or did I miss something?

Are you suggesting that the date and MSGID are both used for dupe
checking by Mystic, and thus because the messages had a different date
(but same MSGID) that's why Mystic let them through?

I believe that's what happened (or it least it would explain what I see at my end). It's not very sophisticated, but it's not wrong. It's unlikely, but MSGIDs might repeat. The creation time of a message must never be modified though.

---
* Origin: . (21:3/102)

deon wrote (2021-03-16):

Re: rescan
By: Oli to deon on Tue Mar 16 2021 10:07 am

Do we know how Mystic's dupe detection work?

Not very well? (again) :P

I know some BBSes use different (additional) methods for dupe detection, but I thought that the "minimum" should have been checking that the MSGID hasnt been used within the last 3 years.

(okay, let's play the blame game)

Right, Msytic should expect modified mails from braindead tossers like hpt and of course every other tosser also should have code that works around husky's fucked-up implementation of a message API. hpt is doing it like this for decades, so every other tosser should have learned that it cannot rely on the creation time of a message :-P

---
* Origin: . (21:3/102)

Re: rescan
By: Oli to deon on Tue Mar 16 2021 02:11 pm

Right, Msytic should expect modified mails from braindead tossers like hpt and of course every other tosser also should have code that works around husky's fucked-up implementation of a message API. hpt is doing it like this for decades, so every
other tosser should have learned that it cannot rely on the creation time of a message :-P

Yeah, I agree - tossers should not modify messages.

And I understand the hpt time issue - But I didnt think that it should be a problem. I know I've read (maybe from you) about it affecting rescans and de-dupe detection - but as I said earlier, I didnt think the message date/time had anything to do with the "minimum" of de-dupe detection, which is based on MSGID. Its probably reasonable that additional checks are done that involve the date/time - and in this case its ineffective with hpt in the path.

Happy for somebody to educate me further, if I'm wrong about the "minimum".

I know SBBS does additional checks - I think one being a CRC on the body (content), so it doesnt matter if the header changes (including the MSGID), the same content will still be caught (which I kinda like).

I've seen some discussion about fixing the time odd/even interval - do you know if that has made it upstream, and if so, I'll update.

...ëîåï

... OUT TO LUNCH - If not back at five, OUT TO DINNER!
--- SBBSecho 3.12-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)