Archive: Site Maintenance
Welcome to acronym heaven. PPP today suffered a DDOS attack. In case that's a term you've never heard before, it means Distributed Denial Of Service attack. With a DDOS attack, the social malcontent responsible usually has access to a few thousand, or even tens of thousands, of computers around the world that will do their bidding. They achieve this by sending out spam email with attachments that people unwittingly open (thereby installing nasty software on their machines) and by uploading files to web servers which again people unwittingly open.
When the ne'er-do-well decides to show the world how truly powerful they are (or, more likely, when they suddenly feel the need to rebel against their actually quite rightful feelings of social inadequacy, ineptitude and general worthlessness), they run up scripts that instruct their small army of compromised machines to simultaneously make requests of a web site. The idea of the game is to take the website down by flooding it with fake traffic.
This particular social dweeb has been attacking PayPerPost since around 4 this morning with a battery of machines all making a request for the same page on the site, over and over any over, many times a second.
Our own defenses have kicked in blocking out the various IP addresses the compromised machines are using, and in some cases even blocking access to entire networks. THis is in order to keep the site up and running. The downside of course is that there are a lot of innocent casualties and as this goes on it's quite likely you may find yourself unable to access PayPerPost.
Once the attack ends (usually when the muppet simply decides to attack someone else, often because the kids parents ground them for not doing the dishes or some other household chore) we'll start to relax the firewall rules again and you should all be able to get into the system with no problem.
Unfortunately these things are now a way of life for many businesses on the Internet and there's very little that can be done about it. So, please forgive the inconvenience while we wait for this particular adolescent with a staggering inferiority complex to proclaim himself victorious to the assorted posters of N-Sync staring down at him from his bedroom walls.
As is our weekly custom, today we making an update to SocialSpark to address some bugs and other issues in the system. The release will require some downtime though (no more than 5 minutes) and so we are making the deployment happen around 23:00 Eastern this evening.
Thanks for your patience.
SocialSpark just went down. We're not seeing any traffic reaching the main application servers and the team are troubleshooting what the cause of that could be.
Update soon.
Update: there was a router failure on one of our cache servers. THis has now been resolved and the site is back online.
We'll be releasing some new features to PayPerPost this morning. The site will not go down this time. The updates to the site will include the following:
- 135 - Fix for a rounding error when calculating cost on opportunity creation.
- 161 - Fix for a rare loophole that allowed some payouts to be sent twice.
- 177 - Navigation changes to help bloggers get back to their reserved opportunity.
- 185 - Fix for the administration feature that crashed while locking blogger accounts.
I don't have any cool screen shots for this one, (sorry). Tonight the site will go down for about 5 - 10 minutes starting at 10:00pm EST while we perform some updates. If you have a reservation that expires at that time, please get your submissions in a few minutes early. The updates to the site will include the following:
- 148 - Proactive Internal Analysis Tool - This is going to help our review team target specific advertisers that we think might be having trouble in the system and reach out to them.
- 151 - Proactive Internal Analysis Tool (part B)
- 169 - Post payout date resetting when a post is resubmitted from initial review stage. This should no longer be a problem after this release.
- 171 - Fix Loophole with "Review My Post" opportunities - These opportunities will now have stricter segmentation so that they only allow the intended recipient to make a post.
As promised we got to the root of the memory leak issue and deployed a fix to the production servers a little while ago.
The site should now be responsive again.
The slowdowns with SocialSpark that we experienced around the same time yesterday are back. The problem seems to be in the messaging system. As lunch hour approaches people are logging on to message their friends and quite a few users have immense friends lists. When you compose a message your friends list is loaded but it's not being removed from memory efficiently when you're done. As a result, our servers have something of a memory leak.
A few of the devs are working hard on a solution to this and we hope to deploy it shortly. Apologies for the inconvenience.
We are experiencing intermittent problems with response times within SocialSpark. We're working on the issue.
Update: We've bounced servers that handle both socialspark and authentication and are watching server load. It's suspected that there's a memory leak somewhere in the chain which is tying up processes.
Tonight the site will go down for 5 - 10 minutes starting at 10:00pm EST while we perform some updates. If you have a reservation that expires at that time, please get your submissions in a few minutes early. The updates to the site will include the following:
- 149 - Master Account - management for advertisers who have multiple accounts. If you are an advertiser with multiple accounts and would like to have this set up, please contact Customer Love by clicking the "Contact Support" link within your account.
- 167 - Update credit card processing - this is self explanatory, we have updated the mechanism behind the scenes that processes credit cards.
- 158 - modify auto approval criteria - we've made the auto approvals smarter by requiring a blogger to have at least 10 posts in the system, and 90% of their posts have to be approved in order to be eligible for this.
- 170 - fix errors thrown when tracking clicks on posts that have been deleted from the system
We're currently experiencing unusually heavy load and a significant drop in response times from SocialSpark. We have team members online looking into this since the load shouldn't have impacted the response times this much.
We will have a resolution very soon.
Update: Issue is now resolved.
