Archive for April, 2010

New Archiving Processes Being Implemented Today

April 30, 2010

In an effort to realize tweets in their appropriate archives more efficiently, a new archiving process has been implemented as of approximately 5:00 am ET (GMT-4) this morning.  During the switch over we fell behind on processing tweets (roughly 150K) but are catching up relatively quickly.

The new process is much more efficient at putting tweets in their appropriate archives (roughly 6x faster) and should keep things running more smoothly.

However, I am also noticing many tweets in the queue that we are receiving from Twitter that are not finding a home in an archive, and I am keeping them to the side for additional investigation over the weekend.  So if you see something amiss in one of your archives, let me know.

~@jobrieniii

John

Planned Ops / Infrastructure Enhancements

April 27, 2010

As outlined in the original post by Brian, we are planning on implementing operational and infrastructure enhancements to help improve the stability and accuracy of the Twapper Keeper service both in the short and long term.

The following slide deck provides insight into the current backend architecture of the service, outlines the challenges we are currently facing as the service scales, and the changes that will be implemented to improve the quality of service to the user community.

The time line for the completion of these activities is 3 weeks, and we will be looking for feedback from the user community as user facing items are implemented.

Join us on Game Changing podcast tonight!

April 26, 2010

Episode 27 – John O’Brien and the TwapperKeeper.

Request to allow tweets to be deleted

April 24, 2010

The User Requests

Over on the UK Web Focus blog Liam Green-Hughes requested that it should be possible for users to delete their tweets from the Twapper Keeper archive. Liam (@liamgh) linked to his comment on Twitter which led to a subsequent discussion of some of the issues: @JulietCulver suggested thatsurely if you write anything on the internet you have to assume it’ll get archived?” and @sputuk (Nick Sharratt) felt thatopting out of archive tweets would be like demanding the right to insist people don’t remember what you say.what you put in public belongs to the public IMO“. Meanwhile @mhawksey (Martin Hawksey) pointed out the middle ground: “I agree with you [@sputuk] but think @liamgh also has a point. The twiter T&C mean I retain IPR for this tweet. Archive needs optout 4 legal“.

Discussion

Although Liam’s subsequent comment that he is “Not convinced of need for archiving” doesn’t reflect the growing interest in the Twitter archiving which has been demonstrated by the Library of Congress’s recent announcement that it will archive all tweets, he also felt that the JISC-funded development work should provide “an opportunity to set a good example“. This is a very pertinent point.

We are also aware that the Twapper Keeper service can provide a valuable role in supporting other value-added services. Martin Hawksey has already provided one demonstration of the value of a Twitter archive with his Twitter captioning service which has already been used to provide captions for videos of Gordown Browns Building Britain’s Digital Future Announcement and JISC 10 plenary talks. But as I pointed out in my post on Twitter and the Digital Economy Bill services such as this could be liable to closure if an overzealous interpretation of the Bill was taken. Developers of such services would, no doubt, be reassured if it was possible to delete their own tweets. And such an approach would avoid the absurdities of having to request each individual person who tweets for permission to reuse their tweets!

Our Plans

Our plan is to provide a way for users to formally request that (1) any tweets that we have archived be deleted and (2) opt’out of any future archiving for their user account.

Initially this deletion and filter will be manually setup for each user after they make the request and a automated filter will be established to filter tweets being archived.

If there is a significant number of people requesting this be done, an automated request form will be established (based upon OAuth login credentials of user).

We’d welcome feedback on these plans.

API Developments to Twapper Keeper

April 19, 2010

As mentioned previously JISC is funding developments to the Twapper Keeper service to reflect the interests of the JISC community – the higher and further education community in the UK. Since such developments are likely to be beneficial to all Twapper Keeper users suggestions for developments are invited from everyone.

Comments to this post should address developments to the Twapper Keeper Web site with suggestions and enhancements invited on the APIs used in conjunction with the Twapper Keeper service.

Note that a previous post invited comments on developments to Twapper Keeper’s user interface.

User Enhancements to Twapper Keeper

April 19, 2010

As mentioned previously JISC is funding developments to the Twapper Keeper service to reflect the interests of the JISC community – the higher and further education communities in the UK. Since such developments are likely to be beneficial to all Twapper Keeper users suggestions for developments are invited from everyone.

Comments to this post should address developments to the Twapper Keeper Web site with suggestions and enhancements invited on the interfaces for the creation of new archives and access to existing Twapper Keeper archives.

The following post will cover API developments.

JISC-Funded Developments To Twapper Keeper

April 16, 2010

The Twapper Keeper service is being used increasingly across across the UK’s higher education sector. Over the past year Twapper Keeper has been used to archive tweets from a number of high profile events including the ALT-C 2009 elearning conference (see archive of #altc2009 tweets),  UKOLN’s Institutional Web Management Workshop (see archive of #iwmw2009 tweets) and, most recently, the JISC’s 21010 conference (see archive of #jisc10 tweets).

JISC (Joint Information Systems Committee) has recognised the importance of Twitter archiving to the higher education community. In order to ensure that the interests of this community are being addressed JISC has provided funding to support further development to Twapper Keeper in order to support developments of interest to this community. The developments will include technical enhancements to the service, support for the sustainability and reliability and the service and the provision of open access to software developments, documentation and lessons learnt and experiences gained.

I am the project manager for this work and will ensure that JISC interests are addressed in this work.  My main dissemination channel  is the UKWebFocus WordPress blog – and as you can see from the Twitter category on the blog, I’ve a long-standing interest in use of Twitter.

As part of the open development process for this work I’ll be publishing a number of posts on this blog which will invite feedback and suggestions for developments to the Twapper Keeper service.  Your comments are welcomed.

Enhance your YouTube vids (and more) with Twitter subtitles!

April 8, 2010

Imagine taking your YouTube (etc.) video and mashing it up with the Twitter feed that happened in real-time about the video.

Now you can with this cool service / api which can now pull tweets from Twitter as well as Twapper Keeper!

Check it out at…

Twitter Powered Subtitles for BBC iPlayer^

1 streaming API + 11 workers + 4 special scripts = 150 Million Tweets Saved

April 2, 2010

This afternoon we passed a major milestone, TwapperKeeper has now saved over 150 MILLION TWEETS.

And other than the disk space issue we have now, the system is running relatively smoothly and keeping up with the throughput of tweets, which often passes 150 tweets / second.

The only major issue we are concerned about is how fast @person archives fill up after first being created – and we will probably introduce a little more advanced logic to our archiving routines so users see those archives fill quickly after creation.

If anyone has any questions, don’t hesitate to ask.

And if you wonder how we are going to pay for all this extra needed disk space, get in line.  We aren’t sure just yet either 😉  A P2P storage concept would be pretty nice about right now… 🙂

@jobrieniii