User:Ragha
Project Title: Language model and Acoustic model for Malayalam language for speech recognition system in CMU Sphinx
What the work is about?
The main objective of speech recognition system is to transcript speech into text. Malayalam is spoken by approximately around 33 million people. Several of the revolutionizing technologies gifted by computer science can be incorporated in people's lives by making this technology more closer and native. One such prominent methods is bringing ease and nativity in modes of communication with computing systems. Speech recognition system contributes towards the same. Speech recognition systems can be broadly classified into two categories based on how they are built. The first type is the one with a limited vocabulary, and in specific a unit is considered to be a single word, which are also known as word recognition system. The second category of systems perform the task of recognition with large vocabulary. The main issue that we face is that prior knowledge of word boundaries are not known to the system. And even if this knowledge is provided, there would be several choices from which the algorithm has to choose the appropriate one, either based on context or statistical likelihood. The idea is that the system needs to check at every instant if that is the word boundary.
SYNOPSIS
Language models that are sophisticated enough to provide adequate context or semantics of the word are required to disambiguate between all the available hypotheses shortlisted. Another issue that needs to be addressed is the effects produced by co-articulatory effects. A particular sound of a word is affected by its predecessor and/or successor. In general natural language conversational speech, such effects are strong. Indic languages provide several instances to show how deep this problem is. The effectiveness of speech recognition relies on 4 primary pointers which are explained in detail on how I plan to address in implementation details. The first one is how the data is handled, i.e size of the vocabulary, speaker independence criterion based on the application needs to be developed. The second one is acoustic modeling which involves the choice of selection of features for each frame or a set of frames. Next is the modeling of language. Finally it is the search problem. Broadly till date, there are two schools of speech recognition technology. They are HMM statistical model and neural net model. I am interested in using statistical model and implement the speech recognition problem using Viterbi algorithm.
IMPLEMENTATON DETAILS
1) Data handling
2) Acoustic modeling
3) Language modeling
4) Search using Viterbi
1) Data Handling and Acoustic Model:
Criteria and choice of selecting an appropriate unit of speech is a very important parameter that determines the quality of speech recognition system. Since Indian languages are syllabic, choice of unit could be syllable. Words are made up of sequence of syllables. S Another issue that the algorithm should address is the co-articulatory effects. With an increase in the vocabulary size, confusion in these aspects increases. Appropriate n gram can be chosen in the HMM model to address this issue, and can further be clustered into equivalence classes. Such context dependent model is to be developed. (most of the systems take trigrams). Firstly, speech input is sampled and pre-processing is done on them to obtain feature vectors for each frame. Sphinx characterizes frame by four features which are cepstra, del(cepstra), del(del(cepstra)), and power. (where del denotes the differential)
The training data includes multiple samples of each word from different speakers so that the system created is independent of the speaker.From implementation point of view, this can be achieved by modeling the constituent syllables that make up the word.
In the training phase, random searches show better results as compared to that of exhaustive search. I am also willing to learn and work upon incorporation of genetic algorithms to optimize this part.
2) Language model: Since explicit word boundaries are not present, the machine would be in a situation to make a selection of output from a large number of word sequence hypotheses. There is a possibility that the alternate hypothesis is syntactically correct. The idea of language model is to select the most likely sequence from all the options the system has. N gram model can be applied here but the issue is memory as large number can be possible. Language modeling can be done in 3 primary ways: 1)Context Free Grammars. But this model is highly restrictive and has to follow the prescription. Hence it is not a good idea to use it in large vocabulary systems. 2)N gram models: An n gram model need not contain the information about the probabilities of all possible n sequences of words. Instead a back off technique, by assigning weights can be applied when the required n gram is not present. 3)Class n gram models: They are similar to the n gram model, with the difference that the tokens are word classes such as months, Named entities, digits etc,. Experiments in the past has shown effective results using trigrams.
Finally, all of it is held on search! Search for the best word sequence given the complete speech input. Viterbi decoding, A* algorithm can be used to implement this task. This can be implemented by processing each frame or a fixed set of frames together and making required updating till that point. This is like time synchronous processing. As discussed earlier, the implementation could be proceeded with stack decoding or dynamic approach. A brief working of stack decoding is explained here.
3) Stack decoding: The possible hypotheses and their respective probabilities are stored. The best hypothesis is thus chosen. If it is complete, then it is the output or else expansion is done for each word and places them in the stack to check further. But this takes a lot of time based on its complexity, so it is better to proceed in dynamic programming based Viterbi approach.
4) Viterbi algorithm: It is based on Hidden Markov Model. The HMM states are traversed in a dynamic programming approach. In a list of dictionaries, the value at 't'th element of 's'th dictionary stores the value of probability corresponding to the best state sequence leading from the initial state at time 0 to state s at time t. In the training phase, the task of classification could be achieved better using SVM(Support Vector Machines). In order to optimize Viterbi, I also propose to use the Elitist model to decide upon the direction during the search phase. I am planning to implement the solution to this problem in Python language. I have also had the experience of building my own POS tagger using Viterbi algorithm. Some of the modules can be adopted here as well.
Let N be the total number of states and T be total duration or the states that are being checked for. Then the complexity of Viterbi decoding is O(N^2T).
Tree structured lexicon: The direct implementation of viterbi decoding still remains expensive. The search space can be optimized by the use of lexical trees based on the fact that for each root, there are several words that share the same prefix. So the model for the system can also be shared.
PROJECT EXECUTION TIME LINE
INITIALIZING WORK AND COMMUNITY BONDING WITH MENTORS: (Attain the knowledge of some specifics in Malayalam language.) (I am having my end semester exams from 19th to 30th April. So it would be preferable to start the work from 2nd May) May 02 - May 06 : Take suggestions from the mentors and incorporating them into the implementation. I also would like to study on SVM to improve pattern recognition accuracies, and if so, I want to discuss witht the mentors to incorporate the changes in the algorithm.
May 07 - May 10 : Design exact details of data structures to be used that are thought of till then to optimally implement the algorithm. Planning out the exact input and output details and intermediate deliverables that are expected.
May 11 - May 18 : Getting feedback from the mentoring community to initiate the implementations.
CODING PERIOD:
May 19 - May 24 : Handle the CMU Sphinx dataset to be worked upon and extract feature vectors for each frame. cepstra, del(cepstra), del(del(cepstra)), and power are to be considered as feature vectors.
May 25 - May 30 : Adapt the exact language model to be opted based on feedback from the mentoring community.
May 31 - June 19 : Implement the Viterbi algorithm to select the best possible word hypotheses for a given input.
June 20 - June 23 : Testing for consistency and efficiency of the models used. Testing()
June 24 - June 27 : Discuss with the mentors regarding the progress of the work to get their feedback and incorporate the suggested changes. Planning to make further error corrections to improve upon the accuracy.
MID TERM DELIVERABLES:
June 28 - July 11 : Develop application specific (if any) that are required to be performed by contacting the mentors.
July 12 - July 20 : Do some proactive further reading if the application or the system could be improved upon in any ways.
July 21 - August 1 : Get suggestions from mentors regarding possible errors and correcting them to improve performance and consistency.
August 2 - August 5 : Documentation of the details of the code. Testing the model repeatedly and correcting errors.
August 5 - August 22 : Backup time for delays not anticipated.
END DELIVERABLE: A comprehensive accomplished speech recognition system for Malayalam language in CMU Sphinx.
Post GSOC : Stay in touch with the mentors and the linguistics community to be a part of the further projects and actively contribute towards the same.
About me
I am a student of IIIT-H(International Institute of Information Technology, Hyderabad). I am an integrated dual degree student currently pursuing B.Tech in Computer Science and MS by Research in Computational Linguistics. I am working under the guidance of Dr.Kishore Prahallad for my research and MS. Currently I have begun studying Deep Neural Network based speech segmentation to work on my research.
Previous experience in the fields of Speech technologies and Computational Linguistics:
1) Earlier I have worked on Text to Speech conversion on mobile platforms for Indian languages, and in specific put effort along Telugu and Kannada, and am also in the process of developing an android application for the same. For improvement in the accuracy and consistency, backoff techniques are implemented.
2) I had the opportunity to learn from Professor Lakshmi Bhai and have worked on the project, Etymological reconstruction of Proto form of Dravidian languages by comparing Malayalam and Telugu.
3) I have previously implemented my own POS tagger based on Viterbi algorithm. Also developed an unsupervised working model of tagging as a part of course project.
4) Developed a plugin for Domain specific morph analysis.
5) Built Intra Chunk Expansion tool for English: Developed a tool to mark intra chunk dependencies of words in English with their expansions from Shakti Standard Format(SSF).
I have submitted a paper on “Domain Adaptation in Morphological Analysis” to ICON-2013: 10th International Conference on Natural Language Processing organized by CDAC. I have also submitted a paper titled “A dynamic programming based approach for generating syllable level templates in statistical parametric synthesis” to IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2014.
Why do you want to work with the Swathanthra Malayalam Computing?
I have always admired and appreciated the contributions of open source development. Moreover due to my mode of study, I have always deeply felt an interest towards linguistically motivated speech technologies. I want to work for SMC as it seems to be the combination of both my passions. I would be extremely enraptured to be able to contribute to Indian languages.
Do you have any past involvement with the Swathanthra Malayalam Computing or another open source project as a contributor?
No. But I am very much interested and love to be a part of it.
Did you participate with the past GSoC programs, if so which years, which organizations?
No
Do you have other obligations between May and August ?
I recognize that GSOC expects a full time commitment. My vacation starts at the beginning of May and classes reopen in the first week of August. We do not have any examinations till the time GSOC coding is officially completed for this year. So I will definitely be able to dedicate 40 hours per week.
Will you continue contributing/ supporting the Swathanthra Malayalam Computing after the GSOC 2014 program, if yes, which area(s), you are interested in?
I definitely would love to be a part of the coding and developing community of SMC even after GSOC. I am interested towards making my work and contributions towards speech related and NLP related problems.
Why should we choose you over other applicants?
The idea of open source has always motivated me. I would passionately work towards making significant contributions. I am very interested in this project and am good in python, the language I intend to use for this project. I am also good at C and Matlab. Most of my previous experience in projects is related to Computational Linguistics, NLP and Speech synthesis. I intend to do my masters and further studies related to speech technologies. So I am sure this would help me a lot and motivate me more towards it.
(A list of my previous projects in this area are mentioned earlier.)
