This post is a after-completion summary of my GSoC project GreenSMW

What was the idea of this project?

The original proposal can be found at http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/nischayn22/1

The main deliverables proposed there were

• Validation of writes using a hash
• Caching of Special Pages
• Identification and caching of frequently made queries in Special:ExportRDF
• Improvements to SMW’s accesses to the database.
• Identification and caching of large inline queries or complex templates using memcache
• Profiling and documentation

What part of this has been achieved, what was left behind?

• Validation of writes using a hash — Done (was very easy to do, got completed in very early)
• Improvements to SMW’s accesses to the database. — Done (this was the most complicated task as it involved lots of refactoring of old code)
• Caching of Special Pages — An alternative strategy is being applied here, the Special page methods are made very efficient and now don’t need any caching as such. (yet to commit this change)
• Identification and caching of frequently made queries in Special:ExportRDF — This was later identified as very low priority as many more places were identified to improve.
• Identification and caching of large inline queries or complex templates using memcache — This task was later identified as not so trivial, memcache uses time based caching, which is not a good solution for query as they involve lots of invalidation. We planned to work on a different technique to invalidate queries by storing their metadata, this is a bigger task and we decided to do it post GSoC. However, users can use a memcache based approach till then as MWJames has been using http://wikimedia.7.n6.nabble.com/Re-Query-result-caching-and-invalidation-Jeroen-De-Dauw-td4981469.html#none
• Profiling and documentation — Mostly done, but more part to be done when SMW 1.8 is going to be released.

What was not in the plan (we don’t have plans for everything, do we?)

• Unit Tests — We covered some parts of SMW’s code using PHPUnit tests.
• Fixed Properties — Side product of re-organizing the DB stuff, wiki admins can assign separate tables for highly used properties so querying takes little time on those.
• Migration Script — A script to let users actually switch to SMW 1.8 without disrupting their site’s activity.
• Semantic diff and site stats — Not fully mature stuff,  but SMW will now be able to produce a diff of the Semantic data, and also store stats of Property usage.

What do you consider the best aspect of participating in GSoC?

The best aspect of participating was contributing to a project that hundreds of people use. Besides, this opportunity gave me immense exposure to the process of Software Development in Open Source

What do you consider the most challenging part of your summer?

Working with existing code was a challenge. I changed something here and it broke something there, such issues occurred many times.

How were your mentors?

Awesome, having two mentors was really beneficial.

Which tips would you give to future students?

Talk to previous year students, talk to mentors as early as possible. Don’t be intimidated by big source codes 😛

What one thing did the Wikimedia community do that you consider very
helpful for your project and would suggest they continue to do?

Developers at Wikimedia have been very helpful throughout, they maintain a friendly atmosphere that welcomes more contributors. I am also thankful to Wikimedia Deutschland for funding my travel to SMWCon in Germany.

Hi, the midterm evaluation has come (time flies doesn’t it?) and finally I am able to put my code to master branch. I had expected this to be a tiresome job with lots of “Git merge conflicts” but thanks to my mentor Markus who suggested to keep the older code along with the newer one to support easier upgradation for users; this led to only copying the new files into a separate folder on the store. Here’s the link to this latest commit of mine https://gerrit.wikimedia.org/r/#/c/14773/

But beware when you read that line on the page below  which says “+6647, -7” . This isn’t true actually :p as the whole lot of code isn’t my work, its mostly inherited code from the older store, with my tweaking at various places. I am looking forward to the next tasks upcoming now as I shall discuss with my mentor, maybe its time for some caching now 🙂

This Update got a little late, but let me sum it up.

New Stuff – UnitTests, DIHandler classes, new table for Booleans

And now the rants start 😉

Before I applied to GSoC I had showed interest in Jeroen’s proposal for writing UnitTests for SMW, now that I chose GreenSMW he is still making me write tests ;), which is actually very useful to test broken stuff rather than our initial attempts of testing by running a wiki. So, when I get stuck for a while waiting for some commit to merge I write tests. With this I am able to make very good use of my “paid time”. I recommend this for all GSoCers 😀

Now back to the Booleans, after a little research on Boolean datatype I finally settled on using TINYINT(1) for MySQL and BOOLEAN for PgSQL, I checked this stuff and it works.  Performance wise we are using very little space for Booleans now and this also makes reading and writing such values faster.

I was suprised that this is not a popular datatype for SMW users after I discovered a bug in SMW’s handling of Booleans.

Also, we discussed about releasing the storerewrite version and though Markus suggested to release it as SQLStore3 keeping also SQLStore2, I and Jeroen pressed on releasing it as just SQLStore, without any redundant code from SQLStore2. There might be a migration script to help users upgrade without any side effects but only if time permits.

Wow its been a month already, time flies!

This week I abstracted (read as separated) the DB layout for SMW. This will make it easier to add new DI types in the future and modify the existing ones. From here follows a series of update to the DB schema.

Till now, as new DI types were added the code was modified to support them *somehow* by using similar naming of fields for most DI types and using some unnecessary fields in many places (Boolean field uses two columns but one is enough and  Geo uses three fields where again one is enough). But now that the DB is abstracted at each DI layer, DB access could be easily modified to enhance performance  for each DI separately and  also merge the DI type String with Blob (will elaborate on this more in future posts).