Archive: Development
Our sites are now slowly coming back online at long last. SocialSpark, IzeaRanks, this blog, the boards all seem to be up and doing just fine. PayPerPost.com is taking a little longer but for no other reason than we're waiting for a DNS server to propogate the changes.
In a nutshell, the data center we used screwed up. I honestly still don't know precisely what happened other than something happened within the data center that knocked everyone there completely off the grid. The servers themselves were never harmed or put in jeopardy and ran along quite happily, just without any traffic. The main Internet feeds in and out of the data center were also fine. Something internally at the data center screwed up and it took them a stunning 10 or so hours to figure it out.
We will absolutely be moving to a distributed architecture as a result of this as soon as possible.
Although there was little we could have done to avoid the problems today I do want to extend my sincere apologies to everyone that was affected by this outage. It's an embarrassment, but one we will learn from and grow as a result of.
The latest release notes are now posted on the boards. Enjoy, and thanks to everyone that provided valuable feedback so that we could address the items in the release today.
We made another fix release of SocialSpark this morning. As always the notes on just what got addressed in the release are over in the launch history section of the Izea Boards.
Managing expectations and perception is probably the most intricate challenge when it comes to opening a site early to a private audience. That's exactly what we did with the Alpha phase of SocialSpark of course. I'm happy with the way the Alpha phase went in that it highlighted a number of areas that we needed to drastically up our game before launch. The downside of course is that old 'perception' chestnut.
The purpose of an Alpha test is to allow a small subset of the actual user base into an application to 'kick the tires'. It gives the dev team great feedback on everything from usability, to performance, to bugs and issues and that need to be fixed before release. We rapidly ramped up the number of invites issued as we neared the end of the Alpha phase as well and as a result got a pretty good view of what the site would be like under load at release time.
Performance was probably the biggest area that we had to address. SocialSpark is really something of a 'mashup' of a number of Izea developed technologies. It uses ITK and IzeaRanks for analytics reporting, RSSBrief for buzz tracking and blog summarization. We developed a financial hub for the application that will later be expanded to support all our apps. The 'ranker' system used by IzeaRanks to synchronize data across our databases and then calculate score and rank for blogs plays a large part in SocialSpark as well. The highly graphical nature of a beautiful site like SocialSpark coupled with a lot of back end interdependencies caused some slow downs that became very evident towards the end of the Alpha. The somewhat brittle nature of some of the connections between the systems also reared it's head on the final day of testing and even on the day of launch when our SocialSpark user base went through the roof.
We have nailed it now though. We learned a lot about scaling a high demand system with PayPerPost, and at times with that system found ourselves fighting a near constant fire of performance and scalability. SocialSpark was designed with the lessons we learned firmly in mind.
We reduced the size of the javascript files we are using by compressing them a few days ago. The CSs files that do all the exquisite layout work on the site are being split and simplified greatly, reducing their size as well. All static content on the site (css, javascripts, images) have been moved onto a content delivery system, exploiting the benefit a CDN gives us in terms of cacheing data but also serving it to you from a server physically near you (we only have the one data center, our CDN has thousands).
SocialSpark's user interface is also made of a bunch of smaller blocks called Components. We undertook some very serious work to work out how long each individual component on a page could be cached without detracting from the near real time functionality of the site. The result is that many components are now only built once per minute, dramatically reducing the load on our application servers. Brittle connections between systems were also reduced and in many cases eliminated, replaced with intricate scripts that are far more resilient to load and far more tolerant of failures.
On the backend, our systems guys did some amazing work building out a hardware architecture that is fault tolerant, fail-over ready and highly scalable. Today the site is running on hardware that gives it 100% growth capacity, and that's from day 1. We'll constantly monitor performance and load on the servers and bring on more hardware as we see growth to constantly maintain a nice comfortable buffer to grow into, rather than scrambling to catch up when load overtakes us and brings the site down.
All in all then, I feel SocialSpark is a great system and technical architecture. While we had some issues leading up to and immediately after launch, the site right now is up, fast, stable and scalable and hopefully providing you all with a fantastic user experience. If you find anything to the contrary though, please don't hesitate to let me know.
It's been gnawing away at you this afternoon hasn't it. You definitely noticed it, but you didn't want to say anything just in case you were the only lucky one out there. It just wouldn't do to draw jealous attention from all the other posties, so you just sat there and enjoyed it, hoping it wasn't some accident that would go away in the near future.
PayPerPost.com is now a heck of a lot faster for everyone. There's a bunch of other people out there that have been thinking the same things you have been thinking today I'm sure. Trevor and Larry have been working tirelessly for the past 2 weeks on a massive rewrite of the segmentation engine to speed the site up and cope with PayPerPost.com's continuing meteoric growth, and today we deployed it.
The end result is, of course, a snappier, happier and all round more pleasant feeling PayPerPost.com than ever before.
Well done guys!
The Alpha Test of IzeaRanks, and tracker engine that feeds it, has gone really well. We’ve had a very high uptake within izearanks.com, and most importantly of all we’ve had some great feedback.
By far the most pressing issue has been with the accuracy of the data that we track and report. Lots of people have said that our numbers are significantly lower than those reported with other analytics tools like Google Analytics and SiteMeter. We’ve worked very hard to identify why this would be the case and come up with solutions. I’m happy to report that the solutions are now live, and you don’t have to do anything at all to update your blog to receive the new script.
First, the previous version of the tools waited for the entire page to load before it started to send us data. However, some of you have blogs that take a while to load. You might have a lot of content for example, or perhaps you have a lot of third party plug ins on your blog, all of which add to the load time. The end result is that visitors to your blog may move on to the next page before the page has finished loading. As a result, the ITK never fires any data to us.
We’ve changed all that. First, we reduced the size of the ITK by 25%. Later today we are also going to compress it, resulting in a reduction of 85% in total from the old version. Second, we changed the code. The new code now fires data in two stages. As soon as the script is loaded, data gets fired at us to record the site visit and page view. Then, when the rest of the page finishes loading the second part fires letting us track activity on your in and outbound links etc. The end result is, of course, more accurate visit recording.
We’ve also got some plans in place to speed up the physical delivery of the Javascript to your readers, but I’ll talk about those in a later blog post.
We also noticed some serious problems with the way Microsoft’s Internet Explorer works with our script. First, Internet Explorer would not recognize the name we gave to the embedded form in your blog. The Javascript creates an embedded form which it populates with the data to fire at the tracker server. Since IE would basically ignore the name we gave the form, the ITK would fail to fire any meaningful data at the server. Obviously this caused the bulk of the low traffic complaints we got, and we’ve fixed how we work with forms in IE now. In addition, we’ve changed the way we batch up the information to send to the server to make it more compatible with blogs that use XHTML instead of plain old HTML. The old way didn’t work too well with XHTML blogs and so, once again, we got lower numbers than we should have.
Finally, on the server we had issues where people with ultra high security settings in their browser caused the ITK to fire blank data at the server. The server, stupidly, didn’t know what to do with the blanks and ignored the entire record. Once again, that meant more low numbers, and it’s been fixed.
I’m still looking at an issue relating to synchronizing blog records between IzeaRanks.com and PayPerPost.com and I’ll be blogging about the work on that when it’s ready to go. For now, I hope you enjoy the changes we’ve made to further improve the ITK and move us a lot closer to getting this thing out of Alpha Test.
Just a quick update. We still have a couple of small issues with the new izearanks.com site that are delaying us sending out information to the people that responded to test it. Of course the whole idea of a test is to find issues, but the remaining problems with the site are related to internal security, so we've got to hold off for just a little while longer.
Jamie is working her tail off today to get the issues resolved and I'll post another update later today as to how far along she gets.
