Twitter Infrastructure Sucks, and Sucks Bad.
May 29, 2008 – 10:34 amAn interesting issue has begun to take the front seat for BeTwittered, our Twitter client for iGoogle. Load on a server is a simple problem. More load requires more CPU, RAM, bandwidth, etc. Very uncomplicated, yes? Now, ignoring other factors, like web server setup and optimization, other services running on the server, etc; what if we complicate this simple equation by throwing in a messaging application? Being ignorant in creating BeTwittered, it never occurred to me that I was creating a messaging application.
I was just creating a client to a messaging service (sure, call Twitter a micro-blog, and I do, but under the covers its a messaging app. It’s not a micro-blog hosting service. That would be easier.) The BeTwittered server gets thousands of requests to send and receive messages per hour. There are over 10k subscribers to BeTwittered, and more than 5k active every day. At current usage, it’s not too bad just handling the load, even with a fairly limited server. The problems get really ugly when Twitter gets slow, or goes down. That happens to be the better part of every day, lately. Instead of client requests coming in, being serviced, and closing, they are just are sitting open on the server. They wait, oh so patiently, as computers and programs can do so well, and slowly pile up and drag the server into a drooling, non responsive state. So, I say, BeTwittered infrastructure sucks, and sucks bad. The title wasn’t a typo, hang in there!
Scott (http://twitter.com/nintender) and I have discussed the solutions a bit. I have two at the moment:
1) Build a Twitter status program that steps in and cuts off communication to Twitter when it gets bad. It would simply check status regularly, and set a flag for the BeTwittered clients letting them know that there is a problem. This would theoretically reduce load by becoming the traffic cop and stopping clients from piling on when traffic gets bad.
2) Build a message queue for BeTwittered clients to submit to. this way there is limited number of communications agents running from BeTwittered to talk to Twitter. No pile up. The trade off would be speed during the heavy loads.
Option 1 is easy. That will probably go in really soon. Option 2 is more complicated, and will probably replace Option 1 when it’s ready.
So why is the title “Twitter Infrastructure Sucks, and Sucks Bad?” I really can’t know, but I can make an educated guess. Twitter appears to be suffering the same problems BeTwittered is suffering, but on a larger scale. With the knowledge I’ve gained writing BeTwittered and dealing with the unexpected popularity, I’m pretty sure that Twitter is now dealing with the same early mis-perception and the fallout that it caused. They didn’t think they were building a messaging infrastructure. Hence, their messaging infrastructure sucks like a turbo-charged shop-vac. No malice, I’m being sarcastic. I feel for them. They’re probably running hard right now trying to adjust.
2 Responses to “Twitter Infrastructure Sucks, and Sucks Bad.”
First of all, thank you for BeTwittered. I use it regularly. I’m sad when it breaks; but when it works, it’s very handy.
As per your blog post here… You could just start building ‘around’ Twitter!
http://webcraftstudios.com/blog/2008/06/27/call-distributed-open-twitter/
If one company were trying to manage all of the email in the world… well, there’s no way. If Twitter style communications are to keep growing, an open model is an imperative.
Take care.
By Zaskoda on Jul 6, 2008
Hi Zakoda,
Thanks for your interesting post. For everyones reference, I left a reply to your post at your site:
http://webcraftstudios.com/blog/2008/06/27/call-distributed-open-twitter/
Regarding BeTwittered, thanks for the feedback. I’m doing my best to start to separate the Twitter outages from the actual BeTwittered “outages” so that you, as a user, can tell when it’s really the twitter service that is out.
Once some strong alternative to twitter comes along, I’ll probably clone BeTwittered for use on the alternate. Could that be open source, and http://identi.ca leading the way?
I’ll also be looking at putting in a back end that uses both/multiple services. this could allow people to ignore which one is down (if there is a connection between them for message flow) or people could just use the new “BeTwittered” for one or multiple services of their choice.
To be clear, there are already alternatives, but from a strict point of view, they are not comparable due to the simple nature, multiple entry points, and ubiquity of Twitter. I’m taking a sort of “purist” standpoint at the moment, because there hasn’t been a real base user shift yet. I think a true, base level, community shift will indicate a better type of service.
In the mean time, there *are* alternatives to the product. For example, Jaiku, Pownce, Friend Feed etc. are alternatives to Twitter. But they are really just “less different” from Twitter than email and instant messaging are. I’m probably splitting hairs a bit, but I can do that. I can afford to sit on the sideline and spout my opinion. I’m just a spectator! :^)
Thanks for the post!
Robert
By robert on Jul 7, 2008