Archive: August 2007
Yesterday we experienced some problems thanks to Googles BlogSpot service going offline.
Today we're seeing some problems with an Amazon service called SQS. When you log in, if your ranks are stale we post a request to SQS which another system here then picks up to refresh your ranks. The SQS queue we use has stopped working.
We're in the process of switching over to a different queue and hope to be back to normal within the hour.
Sorry about this.
Pete
Lincoln was talking about fooling people in his famous quote, but over the years the quote has been contorted to be applicable to customer satisfaction. In it's current version I think it goes something like this
"You can please all of the people some of the time, some of the people all of the time, but you can't please all of the people, all of the time"
In that context at least it's a quote just crying out to be used with software. Over on our boards there's a bunch of people that I'm fairly certain we please all of the time. These are the people who rely on PayPerPost for a supplemental income, an income that would otherwise mean they have to work outside the comfort of their home, pulling some crazy hours doing crazy work just to stay afloat. These are the small business advertisers that want to get some really good exposure on the Internet but who can't financially stomach a huge AdSense or DoubleClick campaign.
I'm fairly certain we make absolutely everyone happy, some of the time. I know the re-introduction of Captcha for example unanimously made pretty much all the posties that had been fighting system cheats happy. I know that when we introduced PPP Direct bloggers and advertisers alike hailed it.
But of course, you can't keep everyone happy forever. Some people love Captcha for example, while others now loathe it. Many people love the country segmentation stuff, but many don't and see it as a way of filtering them out of taking opportunities.
Over on the boards I often see threads asking for new features, for changes, for subtle nuances in the way the system behaves to be fixed. I also see people complaining vehemently about how some aspects of the system are just bad. Some of these are software items, some are infrastructure items (speed and performance on busy days). I've been pushing back quite hard against a lot of requests for change recently, and I wanted to take some time to explain why, and also to explain how we got where we are from a technology standpoint, and where we're going in the future.
PayPerPost was originally developed by one very talented guy working at another company. He was learning Ruby on Rails, the technology that powers PayPerPost, at the time and with that in mind did a stunning job. PayPerPost.com was a very small and simple system back then and the developer did a sterling job keeping up with a huge raft of requests and enhancements coming in from the community and Ted. But, it was always a very reactive way of working, not a proactive one with a lot of up front design. About a month after launching we started to hire the dev team at a rate of 1 or 2 people a month. We grew very rapidly because we needed to. We had an even larger raft of features that needed to be developed, bugs that needed to be fixed, and users that needed supporting. We tried to be proactive with some things and were largely successful, but the majority of work continued to be reactive in nature.
The end result is that PayPerPost.com (the website) experienced some very organic growth. We tried very hard to stay on top of bugs and other issues and at times we didn't do too well. Thankfully today we are well ahead of the bug list. We tried to refactor the code as much as we could as well, striving to maintain as elegant a set of code (elegance in code = ease of maintenance) but the need to continually innovate and maintain our market lead (we are a startup after all) left us on the back foot quite a lot in that arena as well.
While it's a difficult position to be in, it's not unusual. There are literally reams of printed articles and books talking about software entropy as it's known, and how to deal with it. There are tens of thousands of consultants out there that will help gain control of software that has evolved organically, and I used to be one of them. Obviously the goal would be to never arrive in a position where you have to deal with it, but the best laid plans of mice and men....
Earlier this year something wonderful happened. Ted went on vacation. It gave the dev team a chance to step back, really assess where we are at, and where we want to be. It also gave us a chance to just toss around some ideas and try some of them out. In the midst of this I came up with a way to not only pull all those ideas together into a single cohesive system, but also a clear road forward in terms of not only stabilizing the app, but also scaling it to support massive growth.
Ted returned from Europe and was almost instantly dragged into a conference room where myself and two of the senior developers, Trevor and Daniel, sat him down and told him where we thought we should be heading, from a technology perspective.
You know the result of that meeting as "Argus", a name given in honor of Scott's now deceased but much loved dog, but which also was the name given to a mythological hundred-eyed, all knowing, all seeing Titan.
We have 3 clear goals with Argus. The first is obviously competition. There's a feature set in Argus that will just blow your mind. The second goal was to fix those parts of the app that we, the dev team, don't like. The third goal is to make it trivial for us to deliver new features, updates, and fixes in the future. It's not trivial now - very far from it in fact.
The problem is that we need to keep PayPerPost.com as it currently stands running. We can't just abandon the application as it is currently, and go hide in a coccoon until the Argus butterfly emerges. So, the dev team have been driving incredibly hard to get the bug count to zero, and provide a set of quick new features to free us from a lot of the internal support work we have to do. We've also prioritized a set of absolutely vital changes that need to happen to PayPerPost.com fairly rapidly and we've delivered most of those over the past couple of weeks.
The focus now though is well and truly on Argus. We have just 2 months to get together the most amazing overhaul you've ever seen and bind in a bunch of new technology we have that we feel you guys desperately need. For everyone this is a fantastic opportunity to right wrongs, and to make PayPerPost.com everything we always dreamed it would be.
We've done a lot of up front design work, planning and architecture. We've done what should have been done before the site launched in the first place. The end result will be a fast, highly scalable application that can cope with massive growth of the user-base. The resulting application will also be highly maintainable, and uses an architectural design we've come up with that will also allow us to scale up and down the dev team on demand without suffering a productivity hit. We're talking here about a system that is free of bugs, that is snappy to use, that does pretty much everything most of you have asked for it to do, and then some. We've streamlined a number of processes that drive people crazy. We've wiped out unnecessary cruft where we can. We've tightened up the user experience where we've found you guys have a less than optimal experience. Above all else though, we've listened to you. You want hundreds of opps. You want a higher quality network. You want more opportunities to get your brand out (if you're an advertiser) and more ways to make money (if you're a postie). Argus addresses those desires.
We need to remain focussed. We need to keep driving towards our delivery date. We've got a short date because we have to get this done, have to get it out, and have to get you guys using it as fast as reasonably possible. It's everything we all want PayPerPost.com to be, you lot included.
What this means though is that between now and then I have to be ruthless with new features in the existing system. Of course we'll fix bugs. Of course we'll keep providing support to the Customer Love team when we're called on. But as far as new stuff is concerned I have to guard the team. 'Simple' things like adding a link to the boards on all the pages in the app sound trivial, and really they are. But if I give in to one, I have to give in to two, then three, and so on. Every change, however small, also has to go through QA, and then has to be scheduled for a full deployment. The changes also have to be made to somewhat icky source code and then merged down into the Argus source trees. That slows us down, a lot. It's much easier for us just to plan the changes you're asking for into Argus and get them done right. We're also staying a small cohesive team. Every 'small' feature I put a member of the team onto means I lose at least 1/10th of the manpower I have available for Argus.
So, I hope that explains a little of where we are and where we are going. More to the point I hope it explains a little of why I have been so resistant to building in yet more changes between now and November.
But, November's a long way off right? How can you wait that long? I can tell you right now that we will be hand picking between 50 and 100 posties to start beta testing features at the end of September. We'll also be showing a similar number of advertisers. You will get the system you want.
Well it seems even the big boys have issues from time to time. Currently Google's Blogspot.com is offline. This is causing us some problems, or more specifically causing Posties some problems.
When you make a post within the system, PayPerPost.com will automatically hit the blog to check certain criteria in the post. We also validate the URL of the post. With BlogSpot down this is causing PayPerPost.com to experience some timeouts and generally slowing things down (if we have a bunch of people trying to make posts in the system hosted on Blogspot, obviously PayPerPost.com is going to wait a while before it timeouts).
So, for the time being if you are using a BlogSpot.com blog we can't accept your posts. As soon as BlogSpot.com gets back online we'll be fine to keep working with Google Blogspot.com hosted blogs.
Sorry, on Google's behalf.
Pete
UPDATE 2: We will be deploying another set of bugfixes to the servers
tomorrow morning (Aug. 17th) at 9:00 AM EST. The site will be down for about 10 minutes during this time. Again, we apologize for the inconvenience.
UPDATE: We've identified the issue, but it's going to take a bit longer to fix than we originally thought. We're going to keep an eye on it for the next 24 hours.
We've been getting reports that the site is slow or inaccessible for some of our posties, and we're looking into it. We apologize for the inconvenience, and we'll keep you posted on our progress.
Thanks for your patience!
Hey everyone,
Here's a list of the issues that were tackled in our lastest update:
Bloggers will not be assigned a link when looking at an opportunity's detail page, since this was causing some confusion when bloggers would see one link on the detail page a different one when submitting their posts. Instead, you will only receive the link once you have reserved the opportunity.
On that note, you'll notice that the "Take This Opportunity" button has been renamed to "Reserve This Opportunity." If you click this button, you'll be prompted to answer a CAPTCHA in order to reserve the opp. This helps keep automated bots and scripts from reserving opps, and keeps things fair for everyone.
Some new users were having problems receiving confirmation e-mails, and were unable to validate their accounts. This has been fixed.
Once a blogger accepts a PPP Direct offer, they must submit a post within 4 days, or the offer will be canceled automatically. Bloggers will receive e-mails after 2 and 3 days warning them that their offers will be canceled. If an offer is automatically canceled, the blogger and advertiser both receive e-mails letting them know.
The ranking system was returning inaccurate Alexa and PageRank numbers for some blogs. It's been fixed.
Finally, many bloggers were having problems with segmentation locking them out of opportunities at midnight (EST). Many of you night-owls out there will be overjoyed to know that this has been fixed.
Hi all,
We're going to take down PayPerPost.com for around 5 minutes at noon EDT today. This is in order to deploy some bug fixes.
Thanks for your patience.
I wouldn't normally make two posts about the same subject, especially when the beneficiary of the topic is Microsoft (they should be paying us to create an opportunity here!). But there were so many comments on the keyboard post I made yesterday, and emails to me, I felt I had to.
As I suspected, the Microsoft Ergonomic Keyboard 4000 took a lot of getting used to. I found myself throughout the day alternating between using the 'raise' at the front that tilts the keyboard back, and not. It's very odd at first and it feels almost like you are forcing your wrists into the wrist rest with the thing attached. But, my speed at typing on keyboard did gradually increase to near normal speeds by the end of the day, and I did notice when I left the office that I didn't have my usual pain. In fact, I was so happy with no having any wrist pain when I left (it came back when I got in the car though) that I stopped at a store on the way home and bought a second one for my home office.
All in all I've got to say I'm very happy with the device. Yeah it takes a lot of getting used to, even more so than just switching to a split keyboard. The backward 7 degree slope just looks and feels very strange indeed. But the net effect is that the keyboard does do what it's supposed to do, and that's help alleviate the symptons of repetitive strain injuries, well for me it does anyway.
I still miss my DiNovo though.
Oh, one more thing worth pointing out. At home I have a nice desk with a keyboard drawer. I'm not the biggest fan of keyboard drawers, but hey. Anyway, with the 7 degree slope attached, the keyboard only just fits in the drawer and I need to almost pull it out to balance on the edge of the drawer to use it. Be aware of that if you have a similar desk and are thinking of buying this keyboard.
In the past I've had some serious problems with RSI (Repetitive Stress Injury, aka Carpal Tunnel). It usually flares up when I write a book, for obvious reasons, but I've been lucky enough not have any trouble for the past year or so. Well,until yesterday.
Working at home, banging out pages of specs for Argus my left forearm and wrist flared up and I spent the rest of the day and night with a very uncomfortable ache. I have something of a keyboard fetish and will regularly change keyboards for no other reason than I feel the need to. My girlfriend has a similar pocket book fetish.
So, while the RSI flare up would be a great excuse to get a new keyboard, this time I wasn't too happy about it. I love my DiNovo Edge. It goes everywhere with me and is without doubt the single best feeling keyboard I have ever used. It's ultra flat profile though is not good for the typing posture so I reluctantly conceded that I should change it and go for a more 'ergonomic' keyboard. I picked up a Microsoft Natural Ergonomic Keyboard 4000 (what a cheesy name) this morning on the way into the office and I'm just getting used to it now.
It's like a large black alien invaded my desk and it's taking a lot of getting used, thanks to it's ultra weird negative slope. As you can see in the picture above, it slopes away from you at about 7 degrees.
I'm still getting used to it, and missing my DiNovo badly, but already I can see some advantages. My hands are in a more natural position, a less straining one, but also this keyboard has forced me to not to slouch in my chair. I have a tendency to almost lay down in my chair and type on my flat keyboards. That's just not possible with this. You have to sit up to get your head and shoulders above the keyboard in order to not only find the keys, but also to better see what your are doing.
I'll let you know if I still use it in a day or two though.
Jamie has been hounding us. She comes across as so demure and innocent in the Rockstartup videos (while I on the other hand come across like a foul mouthed grinch), but behind the scenes she's a bear. She sends out emails everyday laced with demands for things we need to do, and there's no movement at all on when they can get done.
The latest set have been the "sign up to TypePad and start blogging" emails. The pressure has been intense, and I've resisted as long as I possibly could, but today I had to succumb. I just think I'd crack if she continued to exert that kind of pressure on me. You know, personally, I think she wants my job and is just trying to make me look bad.
So, here I am. A new account, a new blog, and this one's all for the developers. Let the binary mind-melds commence.
