I'm in the process of moving NERP to a new VPS server setup. Just about everything will stay the same for login and mail, but website hosting is moving to a separate VM from the shell host. If you have a website hosted, don't be alarmed, you will be able to login to the new host "stan.nerp.net" soon.
So it turns out when I did some updates to the mail config months ago I somehow removed one of the key spam filtering rules from the server. This has caused a lot more spam than usual hitting people's inboxes and caught spam folders. Thankfully spamassasin was still working to keep most of it out of the inbox. I've re-worked the spam settings and things are looking more "normal" in the logs.
For a couple hours today mail delivery on nerp skipped everyone's procmail filter rules. This allowed custom rules and spam to be sent to the inbox. No mail was lost, but you might have gotten junkmail. There is no need to change anything, the issue has been fixed and filtering is back on.
New on the NERP homepage! You can now login via a SSL encrypted web page. This should work better over firewalled/proxied browsers. Check it out: https://nerp.net/shell
Thanks to Shell in a Box
I upgraded chef.nerp.net to Ubuntu 10.04 LTS today. In addition I added 1.5GB of RAM to the server in order to eliminate the OOM issues that have been making things slow.
I'm investigating a kernel issue with NERP right now that is causing the system to respond slowly. It is a known bug, but I have not found a good patch for the issue yet. I will be rebooting the system later tonight to test some workarounds.
NERP has had another round of system slowness due to mysqld taking over the CPU and kernel resources. I took this downtime to do a system upgrade to the latest Ubuntu which includes updates to mysql and the kernel.
Due to a bug in spamassassin there is a possibility that some mail has been falsely marked as spam. I have fixed the problem but mail sent between new years and Jan 2 01:00 PST will have been affected.
After more failures in the old NERP server I finally gave up and migrated all the data to the new server. The new machine is a VPS hosting system with lots of capacity.
Please let me know ASAP if you are missing any files or software.
A disk in the server had been acting up for a couple days. This morning it got so bad that it crashed and took out the disk controller. This caused damage to the data in the rest of the array. I finally have things mostly fixed, but I will be contacting a few users that have lost data from their home directories. I have some backup data, and will attempt to restore as much as I can.