Wikipedia:Wikipedia Signpost/2012-09-10/Technology report

Technology report

Mmmm, milkshake...

August engineering report published

In August 2012:
  • 97 unique committers contributed patchsets of code to MediaWiki (up by five from June)
  • The total number of unreviewed commits steady at 360.
  • About 35 shell requests were processed (no change).
  • 25 developers received developer access to Git and Wikimedia Labs (down by 55).
  • Wikimedia Labs now hosts 120 projects (up by six), 214 instances (up by three) and 587 users (up by 28).

Engineering metrics, Wikimedia blog

A slide outlining Echo, a new notifications system that would replace watchlists and is already in development

The Wikimedia Foundation's engineering report for August 2012 was published this week on the Wikimedia Techblog and on the MediaWiki wiki, giving an overview of all Foundation-sponsored technical operations in that month (as well as brief coverage of progress on Wikimedia Deutschland's Wikidata project, phase 1 of which is edging its way towards its first deployment). Three of the four headline items in the report have already been covered in the Signpost: the site outage caused by a fibre cut in early August, refinements to the active editors metric, and major work on the Wiki Loves Monuments app, launched last week. The report drew attention to the work of the WMF's Internationalization team on the Universal Language Selector (ULS; see previous Signpost coverage), Project Milkshake to create "generic jQuery components for commonly needed internationalisation features" and WebFonts.

Other items covered in the report include work by WMF Performance Engineer Asher Feldman to expand the number of MySQL servers in the Foundation's secondary data centre Ashburn, and by Tim Starling to write a new Redis-based client for session handling. The two developments are linked insofar as both will be needed for the Foundation to meet its target of making the Virginia site Wikimedia's primary data centre within the next quarter, in an attempt to boost performance. Elsewhere, WMF Security Engineer Chris Steipp worked on adding two new major features to the AbuseFilter extension (global rules and global throttling), for improved detection and prevention of cross-wiki spam. It was, however, a slower month for the TimedMediaHandler, Echo, OAuth and ResourceLoader 2.0 projects.

The monthly report is also a good source of tech news that had otherwise slipped under the radar—in this case, that work on Flow (a talk page reform project) will start in January and that a new mirror for Wikimedia data has been found ("network management solutions" firm), who have also agreed to replicate non-essential data such as "page view files, archives, and more, as well as a full copy of our media files". This month saw the creation of the "Micro Design Improvements" team, an ad-hoc group of staffers who will look at "small but useful design" improvements for MediaWiki, including this week a proposed reworking on the edit window.

Among other news, the first Wikipedia Engineering Meetup (held on 15 August in WMF headquarters in San Francisco), first mentioned in last month's report, attracted approximately 100 developers. The series of two-monthly meetings is intended "to showcase Wikimedia's interesting engineering problems and products to the local developer community"; the inaugural meetup "featured talks about Mobile engineering, Analytics and the VisualEditor".

In brief

Signpost poll
MediaWiki Foundation

Not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for several weeks.

  • The Signpost on your mobile: As covered this week in an editorial, readers with a phone running Google's Android operating system can now read the Signpost on their phones using a dedicated app. The app, developed by former WMF mobile team member (and now volunteer developer) Yuvi Panda, features a mobile-friendly homepage and reading interface, as well as "share" links and a notification system that alerts users when a new edition has been published. The app is available to download for free from Google Play, which is best accessed using the Play Store app on your compatible device.
  • Lua timetable assessed: In light of the largely positive coverage surrounding new template programming language Lua, WMF Engineering Community Manager Sumana Harihareswara took the opportunity to clarify the predicted timetable for deployments. Writing to Wikitech Ambassadors (users who have expressed an interest in spreading tech-related news to their communities), she confirmed that a wider release to Wikimedia wikis was not imminent (being tentatively scheduled this week for early next year) and would be dependent on an assessment in October of the results of the current testing phase. In related news, a user-led study of the "Luaification" of common templates concluded that it was simply too difficult to tell until better tools were in place to separate the effect of the intended changes from environmental confounding factors.
  • HTML5 coming (part III): After three years of speculation and failed attempts to implement it (see previous Signpost coverage), all Wikimedia wikis will be set once more to HTML5 mode on September 17 (wikitech-l mailing list). The news comes two months after WMF Director of Platform Engineering Rob Lanphier expressed renewed interest in the switchover and suggested late July as a possible time to bring it into effect. Developers hope that the move, which should – if everything goes to plan – have little or no visible effect, will this time be met with more permanency than the two previous efforts to future-proof Wikimedia wikis in this way. Since the announcement, the possibility of browsers dropping support for features not in the HTML5 specification (such as font tags) has caused some concern; there is, however, little to suggest any imminent danger, suggesting any decision regarding converting existing usage of such features could be postponed. Any such postponement would of course bring into play the impact of the new wikitext parser, which is unlikely to arrive much before the new year.
  • MediaWikiWidgets.org: is the perfect the enemy of the good? It was suggested this week that MediaWikiWidgets.org, an external site that hosts easy-to-install "widgets" to non-Wikimedia wikis, be merged into MediaWiki.org (wikitech-l). Although the widgets system is widely used, security concerns have meant that it garners little but disdain from a significant percentage of mainstream developers; consequently, views on seemingly legitimising existing widgets (regardless of quality) proved mixed with a clear consensus yet to emerge. An alternative proposal that emerged was the development of a better widgets system to replace the existing security-problem-prone construction.
  • Translation memory upgrade: Translation memory – the use of a substantial back library of translations to inform new ones – is now available to Wikimedia installations of Extension:Translate, rather than just external translation site translatewiki.net, which handles MediaWiki's interface translation (Wikimedia Blog). Developer Niklas Laxström used the opportunity to explain some of the difficulties he encountered in implementing the service on the Wikimedia cluster.
  • Gerrit corruption: The Gerrit-managed code repositories of several extensions and other WMF-hosted projects were temporarily unavailable this week (Wikimedia blog; also wikitech-l). For reasons as yet unknown, a significant number of commits in these repositories were left "dangling": that is to say, important data linking these code revisions to those that preceded and succeeded them was lost. All such cases were later fixed by WMF developer Chad Horohoe, who first managed to prevent the commits being lost forever, and then piece the repositories back together. In a couple of cases, repositories were more seriously corrupted – with committs actively being lost – and are yet to be fully recreated from backups at time of writing. In other news, Wikimedia Labs, which hosts (among other things) a test Gerrit instance, will shortly be dropping its "beta" designation, it was announced this week, promoting some refinements to the way in which accounts are handled.