Outages over the last 24 hours [UPDATED 06/24 @ 10:30 pm ET]

by

As many of you probably noticed, TwapperKeeper has been tough to reach over the last 24 hours.

At first we thought it was a network issue, but after further investigation I believe it is based upon a minor change we made to the apache config file over the weekend which was causing Apache to load up and crash.  (THINK: too much memory allocated to PHP, causing excessive loading, leading to crashes).

Things have been reverted back as of this morning, and we will continue to monitor the site to ensure things continue to run smoother.

One thing to note, even though the site was tough to reach, the archiving processes have continued to run.  We have, however, slowed the post-processing processes which will result in tweets taking a little longer than normal to be realized in their archive.  (This was done during the troubleshooting process and additional jobs will be started shortly to play catch up more quickly.)

[UPDATE 06/24 – 10:30 PM ET] I believe we may have found the issue.  Looks like a code update pushed to production mid-week resulted in many archive queries being run unconstrained, resulting in heavy load in our middle / db tier – and causing things to freewheel “out of control”.  Will continue to monitor through the night.  Archiving processes are running but there is a very large queue of records waiting to be processed.


%d bloggers like this: