CDAC-IDN-Critique

Malayalam IDN Policy Draft by CDAC - Critique by SMC

The Draft policy document by CDAC is available here

File:Response to C-DAC-IDN-Critique.pdf

Criticism on the policy

General Comments

1. The variant table is defined based on random glyphs taken from a list of 900+ possible glyphs for Malayalam. No explanation is given on how two entries in variant table become homo morphs. One entry in variant table is just because of the fact that one is mirror image of other. Since b, d are not excluded from English, there's no need to exclude mirror imaged glyphs in variant table.

   CDAC's Response: The IDN system devised for Malayalam is based only on the modern script. It doesn't address the old script or the fonts based on old script. Also, a detailed study was done before proposing homographs in each of the languages. The study included observing the visual form of the conjunct in the point size of the Address bars of major browsers. The mirror imaged nature of the glyphs was not the criterion for the two glyphs to be qualified as variants.

   Since CDAC took that much effort to check visual spoofing, i just like to know the basic glyph set in modern orthography of Malayalam --Jinesh.k 06:26, 4 December 2010 (PST)

   Please understand that IDN has no relation with the orthography. Orthography is kind of writing style and decided by user's choice on fonts. It is at higher level and close to user, while IDN has nothing to do about the choice of fonts by users. The distinction between Modern Orthography and Old orthography is not technically defined. Unicode or Unicode capable applications never bother about whether user use an Old lipi font or Traditional font. CDAC or IDN policy cannot say that user should use only a certain set of fonts to work with IDN. If CDAC says IDN does not support traditional orthography, it does not make any sense at all for some one who know what is Unicode, Orthography and IDN. - സന്തോഷ് 23:53, 4 December 2010 (PST)

   If mirror image was not a criteria for variant table, explain how ത്സ/ഝ , സ്സ/ഡ്ഡ qualify to the variant table? - സന്തോഷ് 00:00, 5 December 2010 (PST)

   Why browser behavior is studied for drafting IDN standard? Does it mean that if this policy was drafted 10 years, back, none of the Malayalam characters will be allowed in IDN? 10 years back, Malayalam rendering was pathetic in browsers. Or the current policy is going to change when browsers improve the rendering and their address bar behavior later? How can a standard drafted based on its Implementation? - സന്തോഷ് 00:00, 5 December 2010 (PST)

1. Visually identical glyphs are not the only entries to be considered for the variant table. Unicode chart itself has ambiguous dual representations for the same code point without canonical equivalence. An example for this is au signs in Tamil and Malayalam. ௗ- ௌ and ൗ - ൌ . The document does not consider these special cases.

   CDAC's Response: The IDN policy does not permit the entry of syllables having structure CMM or MCM, where M stands for Matra or vowel sign. The ABNF rules takes care of this.

   That is wrong. ൗ - ൌ are neither CMM nor MCM case. It is single code pointed Mathra(vowel signs), appearing with consonants in CM format alone. -സന്തോഷ് 01:04, 5 December 2010 (PST)

2. There are different orthographic forms for many glyphs in Malayalam. The variant table does not address different scenarios arising while considering the visual similarity. For example in traditional orthography TTA is written in stacked form (റ്റ). While in modern orthography it can be written in non-stacked form and this non-stacked form is visually identical to two RA sequence (ററ).

   CDAC's Response : Only the stacked form is considered to be the conjunct TTA in modern orthography.

   i don't really understand the logic. A normal user is easily spoofed with റ്റ and ററ. If we go by CDAC's logic another inorganic standardisation will be the result --Jinesh.k 06:26, 4 December 2010 (PST)

   As noted before, please don't answer questions about IDN using Modern/Old orthography distinction. Nobody with proper Malayalam knowledge will say that nonstacked റ്റ is not TTA. The nonstacked form is explained in detail in Unicode 5.1.0 standard. see http://unicode.org/versions/Unicode5.1.0/ CDACs statement is a contradiction to this.

ABNF rules

1. Section 2 says ക് as pure consonant of ക. Chillu of ക്‍ is considered as pure consonant of ka.

   CDAC's Response: The policy document doesn't address the obsolete characters in the script, although those characters might have been included in Unicode code chart.

   On what basis CDAC decided that a character is obsolete? Does CDAC understand that people write the name "CDAC" in Malayalam using Chillu of K like സിഡാക്‍ ? -സന്തോഷ് 00:20, 5 December 2010 (PST)

2. Section 2.a says CM can be followed by only D (anuswara) or X (visarga). This excludes the Samvruthokarams of Malayalam. All consonant can have cons + u vowel sign + virama and forming samvruthokaram form of that consonant. Examples: തു് , കു് , പു് , രു് .

   CDAC's Response: The use of samvruthokarams is considered to be the part of traditional orthography which the policy doesn't permit.

   We already explained that orthography is not even a subject on discussion here. Can CDAC provide a definition on what is traditional orthography and what is modern orthography? With a list of "allowed" "characters" in modern orthography which is present in traditional Orthography? What is meant by "permitting"? What stops from a user to use a so called modern orthography font to write samvruthokaram? - സന്തോഷ് 00:20, 5 December 2010 (PST)

3. Section 3.a restrict the count of consonant in syllable as 4. But ഗ്ദ്ധ്ര്യ has 5 consonants

   CDAC's Response : Complex conjuncts like ഗ്ദ്ധ്ര്യ have been simplified in modern orthography.

   Just wanted to know how a conjunct can be simplified? If "simplified" will the number of consonants reduce? Will the number of Unicode characters reduce?! CDAC should understand the difference between a conjunct and glyph before answering the question. -- സന്തോഷ് 00:20, 5 December 2010 (PST)

4. Section 3.b excludes syllables with samvruthokram like ക്കു് .
5. Section 4 states a chillu can be followed by a vowel sign. Since chillu is dead consonant, there is no possibility of having virama after chillu.

   CDAC's Response : The document doesn't state that a chillu can be followed by a vowel sign. The observation that a virama can appear after a chillu is based on the recommendation of Unicode

6. The example used for LHC - ന്‍്റ does not exist in printing or digital format. None of the input methods or Malayalam writers ന്റ in this way. The sequence for nta is ന + ് + റ . ie there is no LHC sequence in Malayalam.

   CDAC's Response: The ന്‍്റ happened because none of the rendering engines available today does support the rendering of 'nTa' in the Unicode 5.1 way. as displayed in the document is the wrongly rendered form of the conjunct 'nTa'. This 5.1 official document on rendering the conjunct 'nTa'.

   CDAC said, it did extensive study on rendering in various browser address bars and took policy decision was based on that. So for this, no study was conducted? Never noticed that browsers are not able to render this properly? -സന്തോഷ് 00:57, 5 December 2010 (PST)

7. Since LHC is invalid for Malayalam, including L = ന്‍ , section 5 of the document cancels itself.
8. Because of argument #6, section 6 also cancels itself.
9. Because of arguments #1 to #8 the IDN rule "Consonant Sequence → *3(CH) C [H|D|X|M[D|X]] | L[HC[D|H|M[D]]]" is completely wrong and need to be reformulated.

Restriction Rules

1. Section 2 says "H is not permitted after V, D, X, M, digit and dash" This is wrong since samvruthokaram requires H after V

   CDAC's Response : See the explanation for section 2 under ABNF Rules

2. Section 7 says H can follow L if it is followed by റ , This is wrong as explained above. L can never followed by H. It can only followed by C

   CDAC's Response : See the explanation for section 5 under ABNF Rules

nta criticism

This document does not address the case of stacked and non stacked forms of nta, which are interchangeably used. For example എന്റെ can be spoofed with എന്‍റെ. Severity of this issue is increased by having one more sequence to represent the same conjunct (ന്‍ + ് + റ) is introduced in Unicode 5.1

CDAC's Response : Ans : Modern orthography treats ന്‍റ as 'nRa' and ന്‍റ as 'nTa'. The interchangeable usage of stacked and non-stacked forms for the conjunct 'nTa' is wrong by convention.

Chart of allowed characters

1. Malayalam chillus - the 5.1 version ക്‍ is removed from the tables. which is having same characteristics and use cases of other chillus. So excluding it from the allowed code points does not make any sense. Moreover the existing chillu representation - non-atomic - is not mentioned in the document at all.
2. Malayalam au sign - ൌ is not allowed. Instead the au length mark ൗ is provided. The inscript standard does not allow one to type ൗ and allows only ൌ. Other input methods allows to type both. But the document does not say anything on the equivalence of both. Allowing both vowel signs is also a spoofing issue. And hence this should be handled in variant table.

   CDAC's Response : The inscript standard being revised. The new standard allows both the characters to be inputted. For restricting spoofing and phishing, only one form i.e. ൗ by IDN policy (used in modern text) has been allowed

Variant Table and Visual Spoofing

Variant table is not logical. Only ളള and ള്ള makes sense. None of the other entries should be considered as spoofing. ന്ത and ന്ന is not even close. Mirror images are already used in Latin, eg. b and d. Hence സ്സ and ഡ്ഡ cannot be blocked. Moreover it is not clear why the same logic does not apply for സ and ഡ. It did not consider the case of ററ and non stacked form of റ്റ common in new lipi.

CDAC's Response : The variant table is based on the observations how Malayalam characters and conjuncts are rendered in the address bars of standard browsers like IE, Mozilla and Safari. While ന്ത and ന്ന are perfectly rendered in Mozilla and Safari, they are not legibly rendered in various versions of IE. The mirror imaged nature of the glyphs was not the criterion for the two glyphs to be qualified as variants. Also note that the variant table is not a full-proof mechanism which can prevent spoofing.

Even though similarity is considered, dual encoding is not mentioned. In case of dual encoding of chillus, both forms (atomic chillu and consonant chandrakkala ZWJ) of chillus will look SAME.

CDAC's Response : IDN policy doesn't allow control characters such as ZWJ and ZWNJ to be part of domain names.

Conclusion

The CDAC Policy document on Malayalam IDN is not acceptable without correcting the above explained errors. In its current format, the document was prepared with lot of false assumptions and contains many technical mistakes as pointed out above. Consultation is must with all stake holders of Malayalam Computing before preparing such an important document. This was not happened and we expect such an initiative from the authorities.

Discussion

1. Discussion about the CDACs response: http://lists.smc.org.in/pipermail/discuss-smc.org.in/2010-December/011985.html