Byteplant Forum

Home » CleanMail Support » CleanMail Server Talk » SMTP Service goes down... why?
SMTP Service goes down... why? [message #1360] Mon, 06 February 2006 20:12
iyuvalk
Messages: 26
Registered: December 2004
Junior Member
Hi!

We monitor our NST server with a third-party port monitoring software and we see that the NST server stops responding to SMTP requests for a few seconds then, the server starts responding again, and after a while it goes down again.

What could be the reason for that? the server has 2.5GB of RAM and a 80GB SATA HD.

Thanks,
Yuval Khalifa.
Re: SMTP Service goes down... why? [message #1361 is a reply to message #1360] Tue, 07 February 2006 11:45 Go to previous message
support
Messages: 919
Registered: April 2004
Senior Member
> We monitor our NST server with a third-party port monitoring
> software and we see that the NST server stops responding to
> SMTP requests for a few seconds then, the server starts
> responding again, and after a while it goes down again.
>
> What could be the reason for that? the server has 2.5GB of RAM
> and a 80GB SATA HD.

Please turn on detailed logging and send your nospamtoday.log to
nstsupport@byteplant.com for analysis after such an incident has happened.



Customer Support
Byteplant GmbH
Re: SMTP Service goes down... why? [message #1362 is a reply to message #1360] Tue, 14 February 2006 16:51 Go to previous message
gbh
Messages: 10
Registered: February 2006
Junior Member
i got quite a similar problem. The program stops and log shows `no buffer space available' after running for about 16 hours. It looks like a daily affair. The server has to be restarted to get NST going again.
I have 1GB RAM on W2K sever
Re: SMTP Service goes down... why? [message #1363 is a reply to message #1360] Wed, 15 February 2006 02:24 Go to previous message
Heidner
Messages: 121
Registered: February 2005
Senior Member
How often is the monitoring service touching your server? Do you have a delay filter turned on. If the smtp monitoring service is checking multiple times a minute and you have a 30 second delay filtering enabled -- you could be seeing the monitoring service queries queuing up and using up a connection limit. The NST log files would show you that. Also application and system event logs might give a clue if you see an unusual re-occuring event. Finally if you use task manager and look at the performance screen ... are you seeing memory pressure such that NST is throttling back? (also a setting you can configure).
Re: SMTP Service goes down... why? [message #1364 is a reply to message #1360] Wed, 15 February 2006 12:08 Go to previous message
gbh
Messages: 10
Registered: February 2006
Junior Member
I hav removed delay filter, attachment filter and virus filter after NST crashed the first day. I have only spam filter running.
Event viewer shows event ID 2019, which corresponds to NST log of `no buffer available'.
Task manager shows noSPAMtoday.exe and spamassassin.ex using up almost all of CPU when mails come in.
The mail users connect to server using IMAP, CPU usage is also high.
Re: SMTP Service goes down... why? [message #1365 is a reply to message #1360] Wed, 15 February 2006 12:11 Go to previous message
gbh
Messages: 10
Registered: February 2006
Junior Member
Sorry, it is noSPAMAdmi using up CPU power and not noSPAMToday.exe
Re: SMTP Service goes down... why? [message #1366 is a reply to message #1360] Wed, 15 February 2006 23:07 Go to previous message
Heidner
Messages: 121
Registered: February 2005
Senior Member
Suggestion.... use google and do a search with the phrase "event ID 2019" , lots of hits... some related to Symantec Antivirus (versions 8.0 and earlier).
Re: SMTP Service goes down... why? [message #1367 is a reply to message #1365] Thu, 16 February 2006 11:12 Go to previous message
support
Messages: 919
Registered: April 2004
Senior Member
The 'no buffer space available' corresponds to the 'WSAENOBUFS' error code (10055), which is another keyword you can google for.

Among the first hits you will find a link to this Microsoft knowledge base article:

http://support.microsoft.com/kb/q196271/

It seems there is a registry tweak of the Windows TCP/IP parameters that might solve your problem.



Customer Support
Byteplant GmbH
Re: SMTP Service goes down... why? [message #1368 is a reply to message #1360] Mon, 20 February 2006 10:27 Go to previous message
iyuvalk
Messages: 26
Registered: December 2004
Junior Member
Hi again...

Well, I thought that I solved it by decreasing the number of simultanous connections to the server in the proxy settings dialog, but unfortunately it didn't.

We still see that the NST service is being stopped (in the Event Viewer there's an event that says that the NST Service entered the stopped status.), when I checked the No Spam Today! log file I saw that at the time the service was stopped there's a line tike this:
Feb 14, 2006, 05:21:39 Unhandled exception.
Feb 14, 2006, 05:21:39 No Spam Today! Service V2.3.3.4 1000 Recipients shutting down
Feb 14, 2006, 05:21:39 Statistics load: 366 rows 442 columns
Feb 14, 2006, 05:21:39 Statistics save: 366 rows 442 columns
Feb 14, 2006, 05:21:39 Session 1: Connection from 192.114.186.77 closed
.
.
.
and then all the connections to other hosts are closed and the service stops.

Some configuration details:
The server runs ONLY the NST v2.3.3.4 (and Symantec Antivirus 10.0) and has 2GB RAM and a 3Ghz P4 CPU, this server is located behind a firewall that is connected to the Internet and allows SMTP connections to the server. The NST server delivers the filtered messages to a second proxy (with eSafe Gateway) that cleans the viruses in the messages and then the messages are delivered to Exchange server.

I also checked the Event Viewer and didn't see any new event (regarding the buffer you guys mentioned or any other warnings).

As you can probably understand the subject is *VERY* urgent because when service stops we need to start it manually, and if we don't start it e-mails are blocked.

I'm sorry this message is so long, but I hope that it will help you find the solution to this problem.

Thanks in advance,
Yuval Khalifa.
System R&D.
Mivtach-Simon LTD.


P.S
We had an old server that had only 776MB of RAM and an old CPU and NST v.2.3.3.3 so we upgraded it to server which its details are mentioned above. The funny thing is that although the old server worked slowly it never had this problem.
Re: SMTP Service goes down... why? [message #1369 is a reply to message #1368] Mon, 20 February 2006 10:37 Go to previous message
support
Messages: 919
Registered: April 2004
Senior Member
> Well, I thought that I solved it by decreasing the number of
> simultanous connections to the server in the proxy settings
> dialog, but unfortunately it didn't.
>
> We still see that the NST service is being stopped (in the
> Event Viewer there's an event that says that the NST Service
> entered the stopped status.), when I checked the No Spam Today!
> log file I saw that at the time the service was stopped there's
> a line tike this:
> Feb 14, 2006, 05:21:39 Unhandled exception.
> Feb 14, 2006, 05:21:39 No Spam Today! Service V2.3.3.4 1000
> Recipients shutting down
> Feb 14, 2006, 05:21:39 Statistics load: 366 rows 442 columns
> Feb 14, 2006, 05:21:39 Statistics save: 366 rows 442 columns
> Feb 14, 2006, 05:21:39 Session 1: Connection from
> 192.114.186.77 closed
> .
> .
> .
> and then all the connections to other hosts are closed and the
> service stops.
>
> Some configuration details:
> The server runs ONLY the NST v2.3.3.4 (and Symantec Antivirus
> 10.0) and has 2GB RAM and a 3Ghz P4 CPU, this server is located
> behind a firewall that is connected to the Internet and allows
> SMTP connections to the server. The NST server delivers the
> filtered messages to a second proxy (with eSafe Gateway) that
> cleans the viruses in the messages and then the messages are
> delivered to Exchange server.
>
> I also checked the Event Viewer and didn't see any new event
> (regarding the buffer you guys mentioned or any other
> warnings).
>
> As you can probably understand the subject is *VERY* urgent
> because when service stops we need to start it manually, and if
> we don't start it e-mails are blocked.
>
> I'm sorry this message is so long, but I hope that it will help
> you find the solution to this problem.
>
> Thanks in advance,
> Yuval Khalifa.
> System R&D.
> Mivtach-Simon LTD.
>
>
> P.S
> We had an old server that had only 776MB of RAM and an old CPU
> and NST v.2.3.3.3 so we upgraded it to server which its
> details are mentioned above. The funny thing is that although
> the old server worked slowly it never had this problem.

It could be a hardware problem. Do you have another machine available
to install NoSpamToday! on for a test ?



Customer Support
Byteplant GmbH
Re: SMTP Service goes down... why? [message #1370 is a reply to message #1360] Mon, 20 February 2006 16:31 Go to previous message
iyuvalk
Messages: 26
Registered: December 2004
Junior Member
Hi...

Well, no, I don't have another machine to test it on. Can you be more specific about the hardware problem you think we have? Like, is it the RAM chips? the hard-disks?

I have to say that I find it VERY strange as the server machine is new and works perfectly alright except for this specific issue.

Can it be caused by the new version of NST?

Thanks,
Yuval Khalifa.
Re: SMTP Service goes down... why? [message #1371 is a reply to message #1360] Mon, 20 February 2006 17:37 Go to previous message
Heidner
Messages: 121
Registered: February 2005
Senior Member
I also have the symantec corporate edition antivirus product installed on my box that I'm running NST on. You should visit the Symantec website (if you haven't already) and do a techknowledge query for "Exchange", your version of antivirus. I had found some special notes on directories to specifically exclude from scanning. Then do a similar knowledge query for your version of antivirus software and the event 1019, or no buffers, no connections, socket failure, etc.

This is especially need if you have the corporate version of the antivirus product with the internet security/firewall product. Because it could be periodically trapping out the NST connections.

Did you make the registery tweak the NST support people suggested?
Re: SMTP Service goes down... why? [message #1372 is a reply to message #1360] Tue, 21 February 2006 07:38 Go to previous message
Heidner
Messages: 121
Registered: February 2005
Senior Member
Document ID:2005051215400948
Symantec AntiVirus 10.0 quarantines or deletes Remote Administrator (Radmin)

Document ID:2005040112500448
Best practices for Symantec AntiVirus 10.x Auto-Protect on a Microsoft Exchange server

Document ID:2005082910334448
Error: "The server was unable to allocate from the system paged pool because the pool was empty" on Windows 2003 Server

Description: The server was unable to allocate from the system paged pool because the pool was empty.
Solution: This problem is fixed in Symantec AntiVirus 10.0.2 and Symantec Client Security 3.0.2. (looks like this was documented late August 2005), check your version of Symantec 10, there is a 10.0.2.200 maintenance release available.
Re: SMTP Service goes down... why? [message #1373 is a reply to message #1360] Wed, 22 February 2006 23:29 Go to previous message
RandallRash
Messages: 23
Registered: March 2005
Junior Member
Maybe unrelated, but in this latest version (2.3.3.4), any time I do a File-Apply Settings (Restart Service) it always says it failed to restart service.
Re: SMTP Service goes down... why? [message #1374 is a reply to message #1360] Sun, 26 February 2006 08:17 Go to previous message
iyuvalk
Messages: 26
Registered: December 2004
Junior Member
Hi again...

I regret to tell you that the problem is still here... I removed the Symantec Anti Virus (at least until I see that the problem is completely resolved), The Microsoft registry trick that the NST support suggested has been applied and the EventLog is clean and still I get these message in the NST log file:

Feb 24, 2006, 17:37:09 Unhandled exception.
Feb 24, 2006, 17:37:09 No Spam Today! Service V2.3.3.4 1000 Recipients shutting down
Feb 24, 2006, 17:37:09 Statistics load: 366 rows 442 columns
Feb 24, 2006, 17:37:10 Statistics save: 366 rows 442 columns
Feb 24, 2006, 17:37:10 Session 5: Connection from 192.114.186.77 closed
Feb 24, 2006, 17:37:10 Session 11: Connection from 192.114.186.77 closed
.
.
.

Then the service stops and has to be started manually.
As I mentioned above the server is running on a new PC with 2GB RAM and a 3Ghz P4 CPU. So I don't think that it's a hardware issue (We used the previous version on a much slower PC and I didn't see this problem there.)
At the time of the crash the server was handling about 510 simultaneous connections.

Thanks in advance,
Yuval.

P.S
RandallRash, I also saw that problem a few time on our older PC, usually when you click the Apply Settings again it works fine.
Re: SMTP Service goes down... why? [message #1375 is a reply to message #1360] Sun, 26 February 2006 13:41 Go to previous message
Geforce
Messages: 8
Registered: February 2006
Junior Member
iyuvalk,

personally, I understand why your problem is so urgent, however why don't you just build a standalone PC running something like W2k, install NST and see if that fixes the problem? It could well be down to hardware?

I'm running NST on a PIII 850Mhz with 256mb of PC133 memory, 20gb hard drive. The system is REAL 1990's, some dell crap, but NST works great.

Altenatively, create a Scheduled task to restart the service if the service fails - or tell it to restart the service every 15 minutes Smile

Out of curiosity, why are you running a local virus scanner on the system? - My firewall filters all HTTP/FTP/Socks/Mail etc. So nothing can get in. (Winproxy)

Regards,

Ian Harris.



Post Edited (02-26-06 17:13)
Re: SMTP Service goes down... why? [message #1376 is a reply to message #1360] Sun, 26 February 2006 18:08 Go to previous message
Heidner
Messages: 121
Registered: February 2005
Senior Member
iyuvalk,

Which vesion of Windows are you using again? And the build/service packs that have been installed?

How frequently is this occuring, i.e immediately, every four hours, once a day...

Is there another major application starting up about the same time -- like a backup?

In the service control panel with win2K and later you can specify that services that stop can be automatically restarted. Have you tried changing that setting...

Ian - I would not depend on a single firewall/filtering device to stop all viruses. I have THREE running on the mail server (Clamav, Bitdefender, Symantec Corp Ed) - and each one has stopped a different virus at various times. The problem is that none of the anti virus applications are 100% perfect nor are they always updated immediately. I also specify that clamav is to follow the URL's back in the e-mails so that phishing and e-mails that lead you to infected sites are blocked on the way in.
Re: SMTP Service goes down... why? [message #1377 is a reply to message #1360] Sun, 26 February 2006 18:27 Go to previous message
Geforce
Messages: 8
Registered: February 2006
Junior Member
Thanks for the info..

Quite often I see people's internal mailboxes are "virused" because Norton Antivirus 10.0 picks them up on a different network I have, which would explain why they're not 100% effective.
Re: SMTP Service goes down... why? [message #1378 is a reply to message #1360] Sun, 26 February 2006 20:47 Go to previous message
Heidner
Messages: 121
Registered: February 2005
Senior Member
The other reason for the problems with the virus scans is that when using most e-mail servers you need special versions of anti-virus programs to scan the mail information store. Even for the corporate version of Symantec antivirus - it is an add on option($$).

So for the best protection you need to use multiple approaches:

1) restrict the types of attachments coming in (NST does this nicely)
2) scan incoming mail with a virus scanner (brand A) that is used only for NST mailscanning
3) use a brand B antivirus product to autoprotect and scan the mailhost/NST host
4) use a brand C antivirus product to scan the mail information store.

For brand A - I use the opensource CLAMAV. NST builds a temporary work file that is then passed to CLAMAV for scanning. I have CLAMAV configured to look at the URLS's in the mail and check to see if any of them point to virus or trojans on remote hosts. This remote checking is where I see about half of the incoming "viruses" that CLAMAV stops.

For brand B - I use Symantec Corporate edition - with autoprotect enabled AND exclusions so it does not try to scan the mail store --- this is critical as both symantec and the mail software vendor say that scanning the mailstore can cause corruption. The autoprotect feature catches worms or viruses in about 75% of the NST temporary work files that are created and would normally be passed to CLAMAV... such that CLAMAV doesn't get a file to scan.

For brand C - I use Bitdefender's product to scan the information store. This is a special product that connects to the mail server and actually looks at the content of every e-mail in the database. Occasionally it has caught e-mails that made it past the first two filters - this is typically caused by the lag in the time viruses are released and the virus patterns are available. (I update clamav - daily), symantec twice daily, and bitdefender every eight hours.

Finally I have the corporate edition clients configured with client side mail checking enabled. This means that all outlook clients check the attachements before they are presented to the user for reading or preview.

I have seen one or two viruses caught here... in the last several years. And this is with multiple layers of defense. So it really becomes obvious why you must also keep your WIndows software updated and patched.....
Re: SMTP Service goes down... why? [message #1379 is a reply to message #1375] Mon, 27 February 2006 06:32 Go to previous message
iyuvalk
Messages: 26
Registered: December 2004
Junior Member
Hi again fellers...

As I mentioned before THERE'S NO ANTIVIRUS installed on the NST machine, it runs on a Windows 2003 Server Standard Edition the NST server passes the emails to a second proxy server that scans them for viruses.

This problem occurs about every other week, as for the "Recovery" tab in the Services applet in the Control Panel, that doesn't work either... It only works when the service fails (throws an exception) but here it's the NST that traps an exception and then it closes itself PROPERLY...

The other solution you suggested, to create a scheduled job that restarts the service, say every midnight should work (I guess so...) but isn't there anything better than that?

In order to test the NST on another PC I should explain to my boss why he should buy a new one... and why this one is not good enough. If the NST guys could just tell me that this is a known issue with this type of memory or anything else it would have been much easier...

Thanks for the work you have done so far....
Yuval.
Re: SMTP Service goes down... why? [message #1380 is a reply to message #1360] Wed, 01 March 2006 00:33 Go to previous message
Heidner
Messages: 121
Registered: February 2005
Senior Member
I would not spend money on a new machine - you would most likely have the same problem.

Number one step in trouble shooting is to ask the question what changed that caused the problem. You replaced the server with new hardware - but you probably also either re-installed the OS or changed versions of the OS. My guess is the latter. So the first thing to think about is what type of settings in the OS may be different that could cause the problem. For example are there some kind of housekeeping jobs that are running at night and causing NST to stop.

An example might be a job that stops network services and restarts them for some reason -- because NST is also using network connections it could be getting messed up.

Look for time correlations -- what else might be starting up and perhaps starving NST of resources.

Make sure you have all the logging you can enabled for NST (global settings options).

Look at the event viewer and other application logs to see what might be starting or happening a few minutes before and after the NST service stops.

Do you have any intrution detection software installed on it also (like tripwire), Blackice Server, zonealarm, or even the MS firewall? Check to see if they are recording events. I use Blackice server and occasionally have it stopping apps because I forget to update the baseline... I've also seen MS firewall block stuff that otherwise seems to work most of the time.
Re: SMTP Service goes down... why? [message #1381 is a reply to message #1360] Thu, 02 March 2006 12:11 Go to previous message
gbh
Messages: 10
Registered: February 2006
Junior Member
Yuval, I have similar problem of NST stopping functioning, but the reason is system is out of buffer.
I turned off the NST admin service (green NST icon on desktop) before I went overseas, and surprisingly the problem does not show up for 11 days. When I came back and turned on the NST admin service again, the problem appears.

I have sent logs to NST support for analysis. While waiting for their response, I am seriously considering putting in task schedule to restart the NST admin service every few hours. May be that will ease off the problem.
Re: SMTP Service goes down... why? [message #1382 is a reply to message #1381] Fri, 03 March 2006 12:11 Go to previous message
support
Messages: 919
Registered: April 2004
Senior Member
To avoid double posts, let's continue the discussion here:

http://www.byteplant.com/forum/read.php?f=1&i=1007&t=963

This thread is closed.



Customer Support
Byteplant GmbH
Previous Topic: 3 newbie questions
Next Topic: Installation on Mac OS 10.4
Goto Forum:
  


Current Time: Sun Dec 04 23:22:06 CET 2016