GSoC 2013: Week 9 of Grandham project

In Week 9, I rewrote the complete crowdsourcing functionality of Grandham from a submission based approach to micro-edits based approach. Earlier we used to store the entire snapshot while recording modifications but now we store only the ‘edit’ (similar to diff, but in a structured database table) and pass it to admins for reviewing. This approach has some significant advantages including the easy generation of history of book pages.

In-place editing was integrated during the same week. Now participants can click on a field and edit it.

GSoC 2013: Week 8 of Grandham project

Last weekly I mainly worked on Search feature and admin interfaces. Here’s a quick review:

  • Experimented with various search engines like Sphinx, Ferret, etc and finally decided to use Solr with sunspot gem as it worked well with unicorn data without any configuration.
  • Wrote a general search backend interface for Grandham. We can plugin other search engines anytime as long as it communicated the way Grandham needs it.
  • Added Admin interfaces managing Authors and Publishers.

GSoC 2013: Week 7 of Grandham project

I couldn’t post the weekly updates regularly for the last two times since I was kept more busy than I had expected and the festive holiday that came in between. Here’s a quick review of what happened during the 7th week of GSoC.

  • Grandham got an UI revamp!
  • Integrated new theme.
  • Separate books / authors / publishers lists were implemented
  • Updated user sign in / sign up pages
  • Integrated database clear and fixed failing specs (It’s green now, yay!)

GSoC 2013: Week 6 of Grandham project

This week I’ve been mostly fixing bugs, refactoring code and building admin features. Grandham has built in checks to avoid duplication in Authors/Publishers now.

Submission Review Interface Grandham review page

We have a new interface this week –  A screen to review crowdsourced information submissions. Administrators can use this feature to control the information displayed to user. Currently every review item has 3 actions – Approve, Archive or Comment. Out of these, ‘Comment’ is a new feature which needs to be implemented. It will be helpful for the task force to generate discussions around specific submissions.

Miscellaneous

After a couple of rounds of refactoring, Grandham got a Code Climate score of 4.0 from 1.61! :)

GSoC 2013: Week 5 of Grandham project

Last week, I implemented different parts of the application including the basics of user authorization. Also did a work on refactoring the code base, our tests are clean now.

New features

We now have a page to display information of a specific book, which can be navigated from the books list. This page will eventually contain all the information, including location information and links to download the entire MARC data.

User authorization has been added to Grandham using the gem ‘devise’. Devise provides beautiful forms required for various user-authorization needs, I customized them to work with the look and feel for Grandham. Some of them are here:

Grandham sign in 2Grandham registration

 

 

 

(Social login needs to be implemented in coming weeks)

Our test suite was starting to clutter and due to a problem in factories the CI has been failing randomly. I’ve refactored the test suite and now it all works fine.

Miscellaneous 

In other news, we have switched the browser to run tests from selenium – firefox to poltergeist – phantomjs. It helps us to run tests faster. Also I’ve updated our API and data import script.

GSoC 2013: Week 4 of Grandham project

Last week, I worked on edit book feature and design mockups.

Edit book feature

As you know from previous posts, we have Submission model to store every contribution from users. When an user submits a correction or an edit, it gets stored as a new Submission in submissions table which needs approval from task force to live. ‘Edit’ component of this feature was implented last week.

We have a few routes associated with this feature.

The following route is to post details for adding a new book. It requires language_id (language short code such as ML or EN) to associate the book to that language. Grandham creates new Submission and a new Book instance (with auto generated grandham_id) if submission to this route was successful.

 POST language_submissions_path - /language/:language_id/submissions

Next route is the edit book feature’s route. It requires book_id (grandham_id of the book) to accept information. It creates just a new Submission for the exising book. This submission will be available for task force to approve.

 POST book_submissions_path - /books/:book_id/submissions

Design mockups

Like I mentioned in one of my previous posts, I had drawn the design mockups for basic features of Grandham. The application will be in a state to go live when the features in design mockups are done.

Miscellaneous

We now have a live instance of app running in Heroku – grandham.herokuapp.com.

GSoC 2013: Week 3 of Grandham project

In Week 3, I was mostly designing routes and developing the submission interface for Grandham.

Submission Interface

Grandham

Grandham should have two interfaces for data submission.

  1. Interface for normal users
  2. Interface for advanced users

The first interface, the form to input basic details of a book has been implemented. All the information entered through this form directly goes to ‘Submission’ model with ‘approved’ = false. Later when the task force changes ‘approved’ value to true, all other submission will get disapproved and this submission details would get featured in Book page.

We are storing Author and Publisher information in different models connected to Submission using a has_many :through association and we use the rails feature accepts_nested_attributes_for to input them in the same Submission#new interface.

Miscellaneous

We now have language based navbar in which the links would change according to the language selected. This would help users to quickly navigate to Book, Authors, Publishers and Libraries. It will be an UX enhancement to show selected Language in navbar, will do that.

We have a very basic 2 column books listing now, it’s mainly for development purposes and testing. We will have a proper books listing on the way.

In other news, I got Travis working with selenium integration specs. It required some additional bits of configuration like this –

before_install:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"

GSoC 2013: Week 2 of Grandham project

Week 2 of Grandham was filled with more architecture decisions and implementations.

Solving the versioning problem

It’s quite essential to store different user contributions on the same subject separately if the application supports crowdsourcing. We cannot let the user to overwrite the existing data without any moderation. Hence in Grandham, we have a ‘Book’ model with many ‘Submissions’ out of which one will be an approved submission. A book’s individual page basically has the detail from this approved submission.

Book has many submissions

It will be task force (a set of users with advanced privileges) who would approve submissions. They will have options to comment or even edit a submission.

Miscellaneous

Thanks to Travis CI, Grandham project now has a Continuous Integration server. We also use Gemnasium to track dependency packages.

GSoC 2013: Week 1 of Grandham project

The first week of development which started on 17th of June essentially comprises requirement analysis, planning and API implementation.

Requirement Analysis

Though the basic requirements for Grandham were discussed a month before, it was necessary to have more clarity on the subject. I had a series of discussions with mentors on various aspects of application, especially on how this project would attract information from contributors of different competency. It was sorted out that normal user could provide just the basic information such as Title, Author(s), Publisher(s), Year, Pages, Edition and Description. Advanced users, who are competent with MARC21 format could contribute advanced bibliographic information through a specially designed interface.

As mentioned in the previous post, we had a meeting with Sri K. H. Hussain. Apart from inspiring and guiding us, he made us aware about the importance of having an API and integration with various library systems in existence. I especially remember seeing a live instance of Koha and its super complex MARC21 data input form. It will be really great if it could consume Grandham API and vice versa so that we would have proper data sharing between two information resources.

Planning

I had little confusions around how to store information. I was thinking of extracting basic information from complex MARC21 data to display in individual books page. But since it was decided that users feed them differently, we could easily store basic information separately and use them. Separate data models to store information as Key/Value pair were written, but I removed them all for the sake of readability and easiness. It was this period when we decided to integrate Bootswatch for front end CSS framework.

API Implementation

Books and Fields API were written during period using jbuilder. The data importer script was updated to meet the changes in the application.

Hello Google Summer of Code 2013

GSoC 2013This summer I’m contributing to Swathanthra Malayalam Computing through Google Summer of Code 2013 by developing an application called Grandham, a bibliography data project. I’ve three mentors for this project, namely Anivar Aravind, Mahesh Mukundan and Baiju Muthukadan.

Grandham is essentially a web application to find, contribute and share bibliographic information. It will be serving as an enhanced version of the live application we have now – Malayalagrandham.com. The new application will have support for multiple languages, MARC21 data format, RESTful API, Internationalization and a lot more.

May 27 – June 17 is Community Bonding Period and I’m using this time to discuss and finalize various parts of application with mentors. So far we have decided how the application should collect information from users and how it should organize and store them internally. We need to strictly follow a scalable architecture and avoid internal data duplications.

The project is being built with Ruby on Rails web framework and MySQL as database. The first phase of development includes designing the rest of application and completion of basic API. It’s pretty exciting to build this project, You can follow the development at githubHappy Hacking!