New Archiving Processes Being Implemented Today


In an effort to realize tweets in their appropriate archives more efficiently, a new archiving process has been implemented as of approximately 5:00 am ET (GMT-4) this morning.  During the switch over we fell behind on processing tweets (roughly 150K) but are catching up relatively quickly.

The new process is much more efficient at putting tweets in their appropriate archives (roughly 6x faster) and should keep things running more smoothly.

However, I am also noticing many tweets in the queue that we are receiving from Twitter that are not finding a home in an archive, and I am keeping them to the side for additional investigation over the weekend.  So if you see something amiss in one of your archives, let me know.




2 Responses to “New Archiving Processes Being Implemented Today”

  1. John O'Brien Says:

    As the day has progressed we are starting to fall behind a little (at this time roughly 50k tweets are backlogged) , but I want to see how well it continues to perform and will scale up the number of workers over the weekend to stabilize.

    We have definitely removed the contention with the Streaming API which is now flowing 110+ tweets / sec which will ensure minimize potential for tweet loss during reconnects to Twitter Streaming API.

    (This increased velocity is also why we are quickly getting backlog, as one worker can process approximately 60 tweets / second in its current form. However, the load profile on the server is very nice at this point, so a few more workers and things will be good to go…)


    • John O'Brien Says:

      since we continue to fall a little further behing, 4 new processes have been spawned to catch up. should begin to cut into the tweets in backlog pretty quickly (over the next hour).

Comments are closed.

%d bloggers like this: