User:Govindsmenokee
Google Summer of Code 2013 Proposal for Swathanthra Malalayalam Computing
Building a system and API's for accessing and upadating Malayalamgrandham Bibligiography Data
Personal Information
Email Address ː govindsmenokee@gmail.com Blog URL ː http://blogspotwith.blogspot.in/ Freenode IRC Nick ː govindsmenokee University and current education ː Calicut University, B.Tech Computer Science And Engineering
Why do you want to work with the Swathanthra Malayalam Computing?
Swathanthra Malayalam Computing (SMC) is a free software collective engaged in development, localization, standardization and popularization of various Free and Open Source Softwares in Malayalam language. I would really love to be a part of such an open source community. Localization can help many more people to come into the world wide web scenario.
Do you have any past involvement with the Swathanthra Malayalam Computing or another open source project as a contributor?
As of now no, but i would really love to contribute to an open source project
Did you participate with the past GSoC programs, if so which years, which organizations?
No, I have not participated in any of the previous GsoC programs.
Do you have other obligations between May and August ?
My semester exams end on May 29th. After that I will be able to work 40 hours a week.
Will you continue contributing/ supporting the Swathanthra Malayalam Computing after the GSoC 2013 program, if yes, which area(s), you are interested in?
Yes, I will continue to support the Malayala Grandhavivaram project. My area of interest is Django web developments.
Why should we choose you over other applicants?
I have been a FOSS enthusiast for quite some time and this proposal is for Building a system and API's for accessing and upadating Malayalamgrandham Bibligiography Data which can be done using Django framework as a back-end. I have done my semester project in Django and have quite a good experience in working with Django framework.
My semester project was done in Django 1.5 and is quite similar to this project. So I feel confident that I can do this project.
The Details of my semester project are given below:
GITHUB: https://github.com/govindsmenokee/projectdequeue
ONLINE PORTAL: www.projectdequeue.cz.cc
INTRODUCTION: https://www.youtube.com/watch?v=4EFwVfpUsBk
Proposal Description
An overview of your proposal
This proposal is for the idea “Building a system and API's for accessing and updating Malayalamgrandham Bibliography Data”. www.malayalagrandham.com is a website providing bibliographic information for a large number of books. The earlier version of the webapp was built in a short period of time, due to which it lacks many of the essential features such as dynamic visualization, crowd sourcing MARC21 and MARCXML support, no proper linking between the tables ie no proper ER structure, also there is a need for data validation to ensure the user enters valid data into the fields. This proposal aims at solving all of the problems mentioned.
The following feature will be implemented in the final application:
- Facility for adding/linking copyright expired books to malaylagrandha vivaram
- Adding ISBN & ISBN based seller discovery
- uilding Interface for Publishers through with they can contribute their publication bibliography .
- Similar module for Libraries . That will be added to found in library section of each book
- A module for building qr code of bibliography with a malayalagrandham link
- Crowd sourced way for input and an approval queue interface for submissions.
- MARC21 and MARCXML support
- A dynamic visualisation interface for book information browsing
- proper API , and app work flow documentation.
The implementation is done using Django as a back end and Bootstrap as front end also mysql database will be used to store and retrive data.
The need you believe it fulfills
The website www.malayalagrandham.com will provide users with a option to search for books from a large and growing database. Crowd sourcing will lead to rapid growth in th size of the database. Also API for the app will allow easier use of the service in a number of different websites. Using RESTful API will lead to better code management. The support for mobile devices will bring more number of users to the web application. The qrcodes will help in easier transfer of links to a book information that is stored in a central online repo. The application provides searches based on different parameters like Author, ISBN, Year etc.
Any relevant experience you have
I have done my semester project in Django and successfully completed it. I can easily work with Django framework in building back ends also have a great amount of experience working with the front ends like bootstrap. I run a web design company named WebopX which is a startup at Startup Village, Kochi. I have started working with the existing application, familiarized with the code and made some changes so as to run the application in the newer version of Django. After that started a new project from scratch stripped down the database with only the necessary details and started working with the application. Currently the results from the database are displayed on the web page after pagination limiting 10 result to a page.
The current version of the application can be viewed at my github account, the link is given below:
https://github.com/govindsmenokee/smc-grandham
How you intend to implement your proposal
The proposal for this project is a website with twitter bootstrap as front end and Django as its back end. The data will be stored in a mysql server(either the msqli driver or the mysql driver can be used). Django version 1.4.3 or above can be used for the project.
The main features to be implemented are as follows:
- Facility for adding/linking copyright expired books to malaylagrandha vivaram.
- Adding ISBN & ISBN based seller discovery.
- Building Interface for Publishers through with they can contribute their publication bibliography.
- Similar module for Libraries . That will be added to found in library section of each book.
- A module for building qr code of bibliography with a malayalagrandham link.
- Crowd sourced way for input and an approval queue interface for submissions.
- MARC21 and MARCXML support.
- A dynamic visualisation interface for book information browsing.
- proper API , and app work flow documentation.
Facility for adding/linking copyright expired books to malaylagrandha vivaram
A new field can be added to the existing database in order to mark the copyright status of the book. The duration of the copyright is as follows
Lifetime of the author + until sixty years from the beginning of the calendar year next following the year in which the author dies.
We can now add a field to the authors table which stores the authors year of death. So By simple logic we will be able to categorize books according to the copyright status.
Adding ISBN & ISBN based seller discovery
A field for storing ISBN can be added to the books table. This field can have an Null value in order to encorporate books dont having ISBN numbers.
The change in the schema of the table can be achieved using Django-smuggler. It enables to change the schema of an existing databse without droping it. Also a module named 'south'(http://south.aeracode.org) can be used to make the schema changes without dropping the database or the table itself.
Building Interface for Publishers through with they can contribute their publication bibliography
A seperate class of users can be implemented in the system, these are the verified Publishers, the will have similar capabalities as that of the administrator but only limited to their account ie, they can add and remove the books from their account. They are not allowed to delete other book details but are allowed to modify them. This seperate class of users can be implemented using django-authority plugin.
Similar module for Libraries . That will be added to found in library section of each book
Currently the libraries are just a field in the books table. A separate module for libraries can be built which allows for linking the books to these libraries. The module will allow to add/remove books from specific libraries, it will also enable to locate where a specific book will be available. Alternatively a ManyToManyField can be used to link the books to the libraries.
A module for building qr code of bibliography with a malayalagrandham link
The google charts api can be used to generate qr codes dynamically from a link to the books detailed view page. For example the link to the specific book is www.malayalagrandham.com/books/1 then a qr code can be generated along with the http request through a image tag in the page template withe the image source as:
"http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl=slug_filed". Here the slug field will be the entry in the database containing the address to the page. Please feel free to click the link given above to generate a qr code image.
Crowd sourced way for input and an approval queue interface for submissions
The Crowdsourcing feature is oe of the tough features to implement, there are several methods to achieve this a status field can be added to the approval model which is initially set to 0(not approved or waiting) when the user creates or edits a data. A log for this modification can be stored in a different model with a field as a foriegn key to the status model. A view can be creted based on this log and the admin can change the value of the field status through the interface. All the modified data will be temporarily stored in a separate table and will be periodically flushed according to needs.
Another way of doing this would be to use django-moderation plugin which allows more complex features such as
- moderation queue in admin
- html differences of changes between versions of objects
- configurable email notifications
- custom model form that allows to edit changed data of object
- auto approve/reject for selected user groups or user types
Any of these methods can be used to implement the crowdsourcing feature.
MARC21 and MARCXML support
For reading and writing bibliographic data in MARC21 formats pymarc can be used it easily enables to read and write MARC21 formats. pymarc is a python module so it can be used with django as well.
for example:
>>>from pymarc import MARCReader
>>>reader = MARCReader(open('test/test.dat')) >>>for record in reader:
... print record['24']['a']
the above method can be used to read MARC formats.
Similarly for writing record the following can be used:
>>> from pymarc import Record, Field
>>> record = Record()
>>> record.addField(
... Field(
... tag = '245',
... indicators = ['0','1'],
... subfields = [
... 'a', 'The pragmatic programmer : ',
... 'b', 'from journeyman to master /',
... 'c', 'Andrew Hunt, David Thomas.'
... ]))
>>> out = open('test.dat', 'w')
>>> out.write(record.asMARC21())
>>> out.close()
So by making use of pymarc we can easily read and write in MARC21 formats.
A dynamic visualisation interface for book information browsing
Inorder to build a dynamic interface for book information browsing a Twitter's Bootstrap can be made use of. We can easily create stunning modern interfaces using Bootstrap that also support mobile layouts. Also django-ajax-search can be used to display instant search results and for auto completion this will enhance the user experience.
Proper API and app work flow documentation
Inorder to create an API for the application we can either use the Django-rest-framework or the tastypie framework. Due to better documentation Django-rest-framework is chosen. Django-rest-framework provides option to create RESTful APIs. Django REST framework is a flexible, powerful Web API toolkit.
Using the REST framework we are able to build APIs that are fully self-describing and web browseable. Using the REST framework we can easily create an API for the search feature for the application which can be easily integrated into many other areas. REST framework allows CURD operations. The entire application can be controlled from a terminal using tools like CURL.
REST framework introduces a Request object that extends the regular HttpRequest, and provides more flexible request parsing. We get more methods to pass the data like request.data besides the existing request.post methods.
Using REST framework APIs can be developed for the entire application that are developer friendly.
The final application will have the following features:
- Users can anonymously search for book details.
- Users can opt to register for a new account so as to add data or modify existing data.
- There will be mainly 3 types of users:
1.Normal end user
2.The administrator
3.Verified authors or publishers
- The normal end users can edit the data but it will be queued for admin verification.
- There will be a proper interlinking between the data within the database.
- The administrator can view the data logs and also modify the data using the administration module.
- Fluid grid layouts or browser adaption techniques will be used in order to support mobile devices as majority of the users are having a smart phone and most of the web traffic is created by these devices. It will also be easier to create a browser based app for android, windows phone or any of the other platforms currently available.
- Support for working with bibliographic data encoded in MARC21 using pymarc module.
- Provide option for grouping incomplete tuples so as to easily fill up the missing details.
- Django by default provides i18n support, so the application will have full i18n support. The default i18n can be activated by adding a line into the settings.py file(USE_I18N = True).
- A qrcode will be associated with each book which can be generated from the weblink to that corresponding book.
- Proper APIs will be available that are bulit using the REST framework, will lead to easir integration of the application to many other websites or services.
Initially the login modules will be created, basically 3 types of users will be defined in the application as mentioned earlier.
After the login system is created the administration module can be configured using which data can be visualized and edited.
The following modules and plugins will be used to suppot the apllication:
- Grappelli – Administration (cms)
- Pymarc – For working with MARC21 formats.
- Google chart api – dynamic qrcode generation
- Django-bootstrapped – for enabling bootstrap template
- Django-tagging-ng – for building taxonomy terms within the application
- huey or django-moderation – for implementing the validation queues in administration modules
- Django-pluggables – to produce reusable apps or api
- Django-authority – for creating user types and set permissions
- Django-balancer – this module will only be used if necessary ie, if the size of the database is very large and if the mysql server has large amounts of overloads then this module can be used to split up the data request among different databases.
- Django-smuggler – for manipulating databses from within Django.
- Django-ajax-search - for instant searches and auto complete.
- Django-rest-framework - for implementeing RESTful APIs for the application.
A rough timeline for your progress with phases
| Duration | Description | Mile Stone |
|---|---|---|
| May 29 – June 16 | Before Official Coding Period. Create a basic design and workflow. | Collect all the necessary information regarding the modules and plugins and test them out.
Create a basic structure and workflow for the application and continue to work with the project which is already started(available at https://github.com/govindsmenokee/smc-grandham). |
| June 17 – July 1 | Official Coding Period Starts | Coding,Testing and Debugging of various modules of the the project. The REST framework will be configured with the application and modules will be developed using them in order to create proper APIs for the application.
Implement the modules according to the design and workflow created. implement all the basic back end features such as user registration, user authentication,search, addition of data, deletion, mapping etc Weekly contact and updates with the mentor regarding the project. |
| July 2 - July 20 | Start workin on front end | Start to work on the front end and create a miniature version of the application with all the basic features for the mid term evaluation. |
| July 21 - July 31 | Prepare for mid term evaluation | further enhancements to the application and detection and correction of errors in the application . Make the application perfect for the mid term evaluation.
Disussion with the mentor and addition and deletion of segments from the project. |
| August 1 - August 25 | Advanced features |
Implement the advanced features like MARC21 data import/export, qrcode generation, validation queues, crowd sourcing etc
|
| August 26 - September 6 | UX design and tweaking | Implement the remaining front end design and improve the ux of the appplication tweak the ui so as to incorporate support mobile devices. |
| September 7 - September 13 | Documentation | Test the application for security flaws and make corrections as necessary. Do the documentation for the project. |
| September 14 onwards | Extra time | This is the extra time set aside so as to incorporate any delays in the development life-cycle.
If no delays occur this time can be utilized to further test the application and fix any other bugs. Prepare for the final evaluation. |
Any other details you feel we should consider
It would be better if mobile support for the website is added will draw more number of users towards the application. Also a well defined taxonomy system can be implemented interlinking the terms such as Authors to books and vice-versa If possible Browser based mobile applications should be released onto multiple platforms so as to avoid fragmentation.
Tell us something about you have created
1. http://www.projectdequeue.cz.cc/
(Mirror:http://projectdequeue.webopx.com/projectdequeue.cz.cc/)
This is my semester project which consists of three inter-related components
1.A web server done using drupal core
2.A local server implemented using django
3.and a android application (works with the local server)
the local server which was done in django have some of the features required for this project such as qrcode generation etc.
the link to the local server project is given below:
https://github.com/govindsmenokee/projectdequeue
My github profile is given below find other contributions there
https://github.com/govindsmenokee/
Have you communicated with a potential mentor? If so who?
Yes, I have communicated with the project mentor Anivar Aravind.
