IME handling guide 编辑

This document explains how Gecko handles IME.

Introduction

IME is an abbreviation of Input Method Editor. This is a technical term from Windows but these days, this is used on other platforms as well.

IME is a helper application of a user's text input. It handles native key events before or after focused application (depending on the platform) and creates a composition string (a.k.a. preedit string), suggests a list of what the user attempts to input, commits composition string as a selected item off the list and commits composition string without any conversion. IME is used by Chinese, Japanese, Korean and Taiwan users for inputting Chinese characters because the number of them is beyond thousands and cannot be input from the keyboard directly. However, especially on mobile devices nowadays, IME is also used for inputting Latin languages like autocomplete. Additionally, IME may be used for handwriting systems or speech input systems on some platforms.

If IME is available on focused elements, we call that state "enabled". If IME is not fully available(i.e., user cannot enable IME), we call this state "disabled".

If IME is enabled but users use direct input mode (e.g., for inputting Latin characters), we call it "IME is closed". Otherwise, we call it "IME is open". (FYI: "open" is also called "active" or "turned on". "closed" is also called "inactive" or "turned off")

So, this document is useful when you're try to fix a bug for text input in Gecko.

Composition string and clauses

Typical Japanese IME can input two or more words into a composition string. When a user converts from Hiragana characters to Chinese characters the composition string, Japanese IME separates the composition string into multiple clauses. For example, if a user types "watasinonamaehanakanodesu", it's converted to Hiragana characters, "わたしのなまえはなかのです", automatically (In the following screenshots, the composition string has a wavy underline and the only one clause is called "raw input clause").

When a user presses Convert key, Japanese IME separates the composition string as "わたしの" (my), "なまえは" (name is) and "なかのです" (Nakano). Then, converts each clause with Chinese characters: "私の", "名前は" and "中野です" (In the following screenshot each clause is underlined and not connected adjacently. These clauses are called "converted clause").

If one or more clauses were not converted as expected, the user can choose one of the clauses with Arrow keys and look for the expected result form the list in the drop down menu (In the following screenshot, the clause with the thicker underline is called "selected clause").

Basically, composition string and each clause style is rendered by Gecko. And the drop down menu is created by IME.

Each clause is represented with selection in the editor. From chrome script, you can check it with nsISelectionController. In native code, you can access it with either nsISelectionController or mozilla::SelectionType (the latter is recommended because of type safer). And editor sets these IME selections from mozilla::TextRangeType which are sent by mozilla::WidgetCompositionEvent as mozilla::TextRangeArray. Following table explains the mapping between them.

Note that typically, "Selected clause of raw text typed by the user" isn't used because when composition string is already separated to multiple clauses, that means that the composition string has already been converted by IME at least once.

Modules handling IME composition

widget

Each widget handles native IME events and dispatches WidgetCompositionEvent with mozilla::widget::TextEventDispatcher to represent the behavior of IME in the focused editor.

This is the only module that depends on the users platform. See also Native IME handlers section for the detail of each platform's implementation.

Android widget still does not use TextEventDispatcher to dispatch WidgetCompositionEvents, see bug 1137567.

mozilla::widget::TextEventDispatcher

This class is used by native IME handler(s) on each platform. This capsules the logic to dispatch WidgetCompositionEvent and WidgetKeyboardEvent for making the behavior on each platform exactly same. For example, if WidgetKeyboardEvent should be dispatched when there is a composition is managed by this class in XP level. First of use, native IME handlers get the rights to use TextEventDispatcher with a call of BeginNativeInputTransaction(). Then, StartComposition(), SetPendingComposition(), FlushPendingComposition(), CommitComposition(), etc. are available if BeginNativeInputTransaction() return true. These methods automatically manage composition state and dispatch WidgetCompositionEvent properly.

This is also used by mozilla::TextInputProcessor which can emulates (or implements) IME with chrome script. So, native IME handlers using this class means that the dispatching part is also tested by automated tests.

mozilla::WidgetCompositionEvent

Internally, WidgetCompositionEvent represents native IME behavior. Its message is one of following values:

eCompositionStart

This is dispatched at starting a composition. This represents a DOM compositionstart event. The mData value is a selected string at dispatching the DOM event and it's automatically set by TextComposition.

eCompositionUpdate

This is dispatched by TextComposition when an eCompoitionChange will change the composition string. This represents a DOM compositionupdate event.

eCompositionEnd

This is dispatched by TextComposition when an eCompositionCommitAsIs or eComposiitonCommit event is dispatched. This represents a DOM compositionend event.

eCompositionChange

This is used internally only. This is dispatched at modifying a composition string, committing a composition, changing caret position and/or changing ranges of clauses. This represents a DOM text event which is not in any standards. mRanges should not be empty only with this message.

eCompositionCommitAsIs

This is used internally only. This is dispatched when a composition is committed with the string. The mData value should be always be an empty string. This causes a DOM text event without clause information and a DOM compositionend event.

eCompositionCommit

This is used internally only. This is dispatched when a composition is committed with specific string. The mData value is the commit string. This causes a DOM text event without clause information and a DOM compositionend event.

PresShell

PresShell receives the widget events and decides an event target from focused document and element. Then, it sends the events and the event target to IMEStateManager.

mozilla::IMEStateManager

IMEStateManager looks for a TextComposition instance whose native IME context is same as the widget' which dispatches the widget event. If there is no proper TextComposition instance, it creates the instance. And it sends the event to the TextComposition instance.

Note that all instances of TextComposition are managed by IMEStateManager. When an instance is created, it's registered to the list. When composition completely ends, it's unregistered from the list (and released automatically).

mozilla::TextComposition

TextComposition manages a composition and dispatches DOM compositionupdate events.

When this receives an eCompositionChange, eCompositionCommit or eCompositionCommitAsIs event, it dispatches the event to the stored node which was the event target of eCompositionStart event. Therefore, this class guarantees that all composition events for a composition are fired on same element.

When this receives eCompositionChange or eCompositionCommit, this checks if new composition string (or committing string) is different from the last data stored by the TextComposition. If the composition event is changing the composition string, the TextComposition instance dispatches WidgetCompositionEvent with eCompositionUpdate into the DOM tree directly and modifies the last data. The eCompositionUpdate event will cause a DOM compositionupdate event.

When this receives eCompositionCommitAsIs or eCompositionCommit, this dispatches an eCompositionEnd event which will cause a DOM compositionend event after dispatching eCompositionUpdate event and/or eCompositionChange event if necessary.

One of the other important jobs of this is, when a focused editor handles a dispatched eCompositionChange event, this modifies the stored composition string and its clause information. The editor refers the stored information for creating or modifying a text node representing a composition string.

And before dispatching eComposition* events, this class removes ASCII control characters from dispatching composition event's data in the default settings. Although, this can be disabled with "dom.compositionevent.allow_control_characters" pref.

Finally, this class guarantees that requesting to commit or cancel current composition to IME is perefored synchronously. See Forcibly committing composition section for the detail.

editor/libeditor

mozilla::EditorEventListener listens for trusted DOM compositionstart, text and compositionend events and notifies mozilla::EditorBase and mozilla::TextEditor of the events.

When EditorBase receives an eCompositionStart (DOM "compositionstart") event, it looks for a proper TextComposition instance and stores it.

When TextEditor receives an eCompositionChange (DOM "text") event, it creates or modifies a text node which includes the composition string and mozilla::CompositionTransaction (it was called IMETextTxn) sets IME selections for representing the clauses of the composition string.

When EditorBase receives an eCompositionEnd (DOM "compositionend") event, it releases the stored TextComposition instance.

nsTextFrame

nsTextFrame paints IME selections.

mozilla::IMEContentObserver

IMEContentObserver observes various changes of a focused editor. When an editor or a windowless plugin gets focus, an instance is created, starts to observe and notifies widget of IME getting focus. When the editor or windowless plugin loses focus, it notifies widget of IME losing focus, stops observing everything and is released.

This class observes selection changes (caret position changes), text changes of a focused editor and layout changes (by reflow or scroll) of everything in the document. It depends on the result of nsIWidget::GetIMEUpdatePreference() what is observed.

When this notifies something of widget and/or IME, it needs to be safe to run script because notifying something may cause dispatching one or more DOM events and/or new reflow. Therefore, IMEContentObserver only stores which notification should be sent to widget and/or IME. Then, mozilla::IMEContentObserver::IMENotificationSender tries to send the pending notifications when it might become safe to do that. Currently, it's tried:

• after a native event is dispatched from PresShell::HandleEventInternal()
• at changing focus from a windowless plugin
• when new focused editor receives DOM "focus" event

The 3rd timing may not be safe actually, but it causes a lot of oranges of automated tests.

See also Notifications to IME section for the detail of sending notifications.

Currently, WidgetQueryContentEvent is handled via IMEContentObserver because if it has a cache of selection, it can set reply of eQuerySelectedText event only with the cache. That is much faster than using ContentEventHandler.

e10s support

Even when a remote process has focus, native IME handler in chrome process does its job. So, there is process boundary between native IME handler and focused editor. Unfortunately, it's not allowed to use syncronous communication from chrome process to a remote process. This means that chrome process (and also native IME and our native IME handler) cannot query the focused editor contents directly. For fixing this issue, we have ContentCache classes around process boundary.

mozilla::ContentCache

This is a base class of ContentCacheInChild and ContentCacheInParent and IPC-aware. This has common members of them including all cache data:

mText

Whole text in focused editor. This may be too big but IME may request all text in the editor.

If we can separate editor contents per paragraph, moving selection between paragraphs generates pseudo focus move, we can reduce this size and runtime cost of ContentEventHandler. However, we've not had a plan to do that yet. Note that Microsoft Word uses this hack.

mCompositionStart

Offset of composition string in mText. When there is no composition, this is UINT32_MAX.

mSelection::mAnchor, mSelection::mFocus

Offset of selection anchor and focus in mText.

mSelection::mWritingMode

Writing mode at selection start.

mSelection::mAnchorCharRect, mSelection::mFocusCharRect

Next character rect of mSelection::mAnchor and mSelection::mFocus. If correspoinding offset is end of the editor contents, its rect should be caret rect.

These rects shouldn't be empty rect.

mSelection::mRect

Unifiied character rect in selection range. When the selection is collapsed, this should be caret rect.

mFirstRect

First character rect of mText. When mText is empty string, this should be caret rect.

mCaret::mOffset

Always same as selection start offset even when selection isn't collappsed.

mCaret::mRect

Caret rect at mCaret::mOffset. If caret isn't actually exists, it's computed with a character rect at the offset.

mTextRectArray::mStart

If there is composition, mStart is same as mCompositionStart. Otherwise, UINT32_MAX.

mTextRectArray::mRects

Each character rects of composition string.

mEditorRect

The rect of editor element.

mozilla::ContentCacheInChild

This exists only in remote processes. This is created as a member of PuppetWidget. When PuppetWidget receives notifications to IME from IMEContentObserver in the remote process, it makes this class modify its cached content. Then, this class do that with WidgetQueryContentEvents. Finally, PuppetWidget sends the notification and ContentCacheInParent instance as ContentCache to its parent process.

mozilla::ContentCacheInParent

This exists as a member of TabParent. When TabParent receives notification from corresponding remote process, it assigns ContentCacheInParent new ContentCache and post the notification to ContentCacheInParent. If all sent WidgetCompositionEvents and WidgetSelectionEvents are already handled in the remote process, ContentCacheInParent sending the notifications to widget.

And also this handles WidgetQueryContentEvents with its cache. Supported event messages of them are:

• eQuerySelectedText (only with SelectionType::eNormal)
• eQueryTextContent
• eQueryTextRect
• eQueryCaretRect
• eQueryEditorRect

Additionally, this does not support query content events with XP line breakers but this must not be any problem since native IME handlers query contents with native line breakers.

ContentCacheInParent also manages sent WidgetCompositionEvents and WidgetSelectionEvents. After these events are handled in the remote process, TabParent receives it with a call of RecvOnEventNeedingAckHandled(). Then, it calls ContentCacheInParent::OnEventNeedingAckHandled(). Finally, ContentCacheInParent flushes pending notifications.

How do mozilla::TextComposition and mozilla::IMEStateManager work in e10s mode?

In remote process, they work as non-e10s mode. On the other hand, they work specially in parent process.

When IMEStateManager in parent process receives  eCompositionStart, it creates TextComposition instance normally. However, if the event target has remote contents, TextComposition::DispatchCompositionEvent() directly sends the event to the remote process instead of dispatching the event into the target DOM tree in the process.

That means that even in a parent process, anybody can retrieve TextComposition instance, but it just does nothing in parent process.

IMEStateManager works more complicated because IMEStateManagers in each processe need to negotiate about owner ship of managing input context.

When a remote process gets focus, temporarily, IMEStateManager in parent process disables IME in the widget. After that, IMEStateManager in the remote process will set proper input context for the focused editor. At this time, IMEStateManager in the parent process does nothing. Therefore, IMEContentObserver is never created while a remote process has focus.

When a remote process loses focus, IMEStateManager in parent process notifies IMEStateManager in the remote process of "Stop IME state management". When IMEStateManager::StopIMEStateManagement() is called in the remote process by this, the IMEStateManager forgets all focus information (i.e., that indicates nobody has focus).

When IMEStateManager in parent process is notified of pseudo focus move from or to menubar while a remote process has focus, it notifies the remote process of "Menu keyboard listener installed". Then, TabChild calls IMEStateManager::OnInstalledMenuKeyboardListener() in the remote process.

Style of each clause

The style of each IME selection is managed by LookAndFeel class per platform. Therefore, it can be overridden by prefs.

Background color, foreground color (text color) and underline color can be specified with following prefs. The values must be string of "#rrggbb" format.

• ui.IMERawInputBackground
• ui.IMERawInputForeground
• ui.IMERawInputUnderline
• ui.IMESelectedRawTextBackground
• ui.IMESelectedRawTextForeground
• ui.IMESelectedRawTextUnderline
• ui.IMEConvertedTextBackground
• ui.IMEConvertedTextForeground
• ui.IMEConvertedTextUnderline
• ui.IMESelectedConvertedTextBackground
• ui.IMESelectedConvertedTextForeground
• ui.IMESelectedConvertedTextUnderline

Underline style can be specified with the following prefs. The values are integer,  0: none, 1: dotted, 2: dashed, 3: solid, 4: double, 5: wavy (The values same as NS_STYLE_TEXT_DECORATION_STYLE_* defined in nsStyleConsts.h.

• ui.IMERawInputUnderlineStyle
• ui.IMESelectedRawTextUnderlineStyle
• ui.IMEConvertedTextUnderlineStyle
• ui.IMESelectedConvertedTextUnderlineStyle

Underline width can be specified with "ui.IMEUnderlineRelativeSize" pref. This affects all types of clauses. The value should be 100 or 200. 100 means normal width, 200 means double width.

On some platforms, IME may support its own style for each clause. Currently, this feature is supported in TSF mode of Windows and on Linux. The style information is stored in TextRangeStyle which is defined in TextRange.h. It's a member of TextRange. TextRange is stored in mRanges of WidgetCompositionEvent only when its message is eCompositionChange.

Lifetime of composition string

When native IME notifies Gecko of starting a composition, a widget dispatches WidgetCompositionEvent with eCompositionStart which will cause a DOM compositionstart event.

When native IME notifies Gecko of a composition string change, a caret position change and/or a change of length of clauses, a widget dispatches WidgetCompositionEvent with eCompositionChange event. It will cause a DOM compositionupdate event when composition string is changing. That is dispatched by TextComposition automatically. After that when the widget and PresShell of the focused editor have not been destroyed yet, the eCompositionChange will cause a DOM text event which is not in any web standards.

When native IME notifies Gecko of the ending of a composition, a widget dispatches WidgetCompositionEvent with eCompositionCommitAsIs or eCompositionCommit. If the committing string is different from the last set of data (i.e., if the event message is eCompositionCommit), TextComposition dispatches a DOM compositionupdate event. After that, when the widget and PresShell of the focused editor have not been destroyed yet, an eCompositionChange event dispatched by TextComposition, that causes a DOM text event. Finally, if the widget and PresShell of the focused editor has not been destroyed yet too, TextComposition dispatches an eCompositionEnd event which will cause a DOM compositionend event.

Limitation of handling composition

Currently, EditorBase touches undo stack at receiving every WidgetCompositionEvent. Therefore, EditorBase requests to commit composition when the following cases occur:

• The editor loses focus
• The caret is moved by mouse or Javascript
• Value of the editor is changed by Javascript
• Node of the editor is removed from DOM tree
• Somethings object is modified in an HTML editor, e.g., resizing an image
• Composition string is moved to a different position which is specified by native IME (e.g., only a part of composition is committed)

In the future, we should fix this limitation. If we make EditorBase not touch undo stack until composition is committed, some of the cases must be fixed.

Notifications to IME

XP part of Gecko uses nsIWidget::NotifyIME() for notifying widget of something useful to handle IME. Note that some of them are notified only when nsIWidget::GetIMEUpdatePreference() returns flags which request the notifications.

NOTIFY_IME_OF_TEXT_CHANGE, NOTIFY_IME_OF_SELECTION_CHANGE, NOTIFY_IME_OF_POSITION_CHANGE and NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED are always sent by following order:

1. NOTIFY_IME_OF_TEXT_CHANGE
2. NOTIFY_IME_OF_SELECTION_CHANGE
3. NOTIFY_IME_OF_POSITION_CHANGE
4. NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED

If sending one of above notifications causes higher priority notification, the sender should abort to send remaning notifications and restart from highet priority notification again.

Additionally, all notifications except NOTIFY_IME_OF_BLUR should be sent only when it's safe to run script since the notification may cause querying content and/or dispatching composition events.

NOTIFY_IME_OF_FOCUS

When an editable editor gets focus and IMEContentObserver starts to observe it, this is sent to widget. This must be called after the previous IMEContentObserver notified widget of NOTIFY_IME_OF_BLUR.

Note that even if there are pending notifications, they are canceled when NOTIFY_IME_OF_FOCUS is sent since querying content with following notifications immediately after getting focus does not make sense. The result is always same as the result of querying contents at receiving this notfication.

NOTIFY_IME_OF_BLUR

When an IMEContentObserver instance ends observing the focused editor, this is sent to widget synchronously because assumed that this notification causes neither query content events nor composition events.

If widget wants notifications even while all windows are deactive, IMEContentObserver doesn't end observing the focused editor. I.e., in this case, NOTIFY_IME_OF_FOCUS and NOTIFY_IME_OF_BLUR are not sent to widget when a window which has a composition is being activated or deactivated.

When widget wants notifications during deactive, widget includes NOTIFY_DURING_DEACTIVE to the result of nsIWidget::GetIMEUpdatePreference().

If this notification is tried to sent before sending NOTIFY_IME_OF_FOCUS, all pending notifications and NOTIFY_IME_OF_BLUR itself are canceled.

NOTIFY_IME_OF_TEXT_CHANGE

When text of focused editor is changed, this is sent to widget with a range of the change. But this is sent only when result of nsIWidget::GetIMEUpdatePreference() includes NOTIFY_TEXT_CHANGE.

If two or more text changes occurred after previous NOTIFY_IME_OF_TEXT_CHANGE or NOTIFY_IME_OF_FOCUS, the ranges of all changes are merged. E.g., if first change is from 1 to 5 and second change is from 5 to 10, the notified range is from 1 to 10.

If all merged text changes were caused by composition, IMENotification::mTextChangeData::mCausedOnlyByComposition is set to true. This is useful if native IME handler wants to ignore all text changes which are expected by native IME.

If at least one text change of the merged text changes was caused by current composition, IMENotification::mTextChangeData::mIncludingChangesDuringComposition is set to true. This is useful if native IME handler wants to ignore delayed text change notifications.

If at least one text change of the merged text changes was caused when there was no composition, IMENotification::mTextChangeData::mIncludingChangesWithoutComposition is set to true.

NOTIFY_IME_OF_SELECTION_CHANGE

When selection (or caret position) is changed in focused editor, widget is notified of this.

If the last selection change was occurred by a composition event event handling, IMENotification::mSelectionChangeData::mCausedByComposition is set to true. This is useful if native IME handler wants to ignore the last selection change which is expected by native IME.

If the last selection change was occurred by an eSetSelection event, IMENotification::mSelectionChangeData::mCausedBySelectionEvent is set to true. This is useful if native IME handler wants to ignore the last selection change which was requested by native IME.

If the last selection is occurred during a composition, IMENotification::mSelectionChangeData::mOccurredDuringComposition is set to true. This is useful if native IME handler wants to ignore the last selection change which occurred by web application's compositionstart or compositionupdate event handler before inserting composition string.

NOTIFY_IME_OF_POSITION_CHANGE

When reflow or scroll occurs in the document, this is sent to widget. But this is sent only when result of nsIWidget::GetIMEUpdatePreference() includes NOTIFY_POSITION_CHANGE.

This might be useful to update a candidate window position or something.

NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED

After TextComposition handles eCompositionStart, eCompositionChange, eComposiitionCommit or eCompositionCommitAsIs, this notification is sent to widget. This might be useful to update a candidate window position or something.

NOTIFY_IME_OF_MOUSE_BUTTON_EVENT

When a mousedown event or a mouseup event is fired on a character in a focused editor, this is sent to widget. But this is sent only when result of nsIWidget::GetIMEUpdatePreference() includes NOTIFY_MOUSE_BUTTON_EVENT_ON_CHAR. This is sent with various information. See IMENotification::mMouseButtonEventData in IMEData.h for the detail.

If native IME supports mouse button event handling, widget should notify IME of mouse button events with this. If IME consumes an event, widget should return NS_SUCCESS_EVENT_CONSUMED from nsIWidget::NotifyIME(). Then, nsEditor doesn't handle the mouse event.

Note that if a mousedown event or a mouseup event is consumed by a web application (before a focused editor handles it), this notification is not sent to widget. This means that web applications can handle mouse button events before IME.

Requests to IME

XP part of Gecko can request IME to commit or cancel composition. This must be requested via IMEStateManager::NotifyIME(). Then, IMEStateManager looks for a proper TextComposition instance. If it's found, TextComposition::RequestToCommit() for calling nsIWidget::NotifyIME() and handles some extra jobs.

widget should call the proper native API if it's available. Even if commit or canceling composition does not occur synchronously, widget doesn't need to emulate it since TextComposition will emulate it automatically. In other words, widget should only request to commit or cancel composition to IME.

REQUEST_TO_COMMIT_COMPOSITION

A request to commit current composition to IME. See also following "Forcibly committing composition" section for additional information.

REQUEST_TO_CANCEL_COMPOSITION

A request to cancel current composition to IME. In other words, a request to commit current composition with an empty string.

Forcibly committing composition

When TextComposition::RequestToCommit() calls nsIWidget::NotifyIME(), it guarantees synchronous commit or canceling composition.

In order to put it into practice, we need to handle the following four scenarios:

The composition is committed with non-empty string synchronously

This is the most usual case. In this case, TextComposition handles WidgetCompositionEvent instances during a request normally. However, in a remote process in e10s mode, this case never occurs since requests to native IME is handled asynchronously.

The composition is not committed synchronously but later

This is the only case in a remote process in e10s mode or occurs on Linux even in non-e10s mode if the native IME is iBus. The callers of NotifyIME(REQUEST_TOCOMMIT_COMPOSITION) may expect that composition string is committed immediately for their next job. For such a case, TextComposition::RequestToCommit() synthesizes DOM composition events and a DOM text event for emulating to commit composition synchronously. Additionally, TextComposition ignores comming events which are dispatched by widget when the widget receives native IME events.

In this case, using the last composition string as commit string.

However, if the last composition string is only an ideographic space (fullwidth space), the composition string may be a placeholder of some old Chinese IME on Windows.

In this case, although, we should not commit the placeholder character because it's not a character which the user wanted to input but we commit it as is. The reason is, inputting an ideographic space causes a composition. Therefore, we cannot distinguish if committing composition is unexpected. If the user uses such old Chinese IME, "intl.ime.remove_placeholder_character_at_commit" pref may be useful but we don't support them anymore in default settings (except if somebody will find a good way to fix this issue).

The composition is committed synchronously but with empty string

This case may occur on Linux or with some IME on other platforms. If a web application implements autocomplete, committing with different strings especially an empty string it might cause confusion.

In this case, TextComposition overwrites the commit string of eCompositionChange event dispatched by widget. However, if the last composition string is only an ideographic space, it shouldn't be committed. See the previous case

Note that this case doesn't work as expected when composition is in a remote process in e10s mode.

The composition is not committed

On Linux, there is no API to request commit or canceling composition forcibly. Instead, Gecko uses gtk_im_context_reset() API for this purpose because most IME cancel composition with it. But there are some IMEs which do nothing when Gecko calls it.

If this occurs, Gecko should restart composition with a DOM compositionstart event , a DOM compositionupdate event and a DOM text event at caret position.

This issue hasn't been supported yet.

IME state management

IME is a text input system. It means that except when a user wants to input some text, IME shouldn't be avaliable. For example, pressing the space key to attempt scrolling a page may be consumed and prevented by IME. Additionally, password editors need to request special behavior with IME.

For solving this issue, Gecko sets the proper IME state at DOM focus change.

First, when a DOM node gets focus, nsFocusManager notifies IMEStateManager of the new focused node (calls IMEStateManager::OnChangeFocus()). IMEStateManager asks desired IME state by calling nsIContent::GetDesiredIMEState() of the node. If the node owns nsEditor instance, it asks for the desired IME state from the editor and returns the result.

Next, IMEStateManager initializes InputContext (defined in IMEData.h) with the desired IME state and node information. Then, it calls nsIWidget::SetInputContext() with the InputContext.

Finally, widget stores the InputContext and enables or disables IME if the platform has such an API.

InputContext

InputContext is a struct. Its mIMEState, mHTMLInputType, mHTMLInputInputMode and mActionHint are set at nsIWidget::SetInputContext() called.

mIMEState

IME state has two abilities. One is enabled state:

ENABLED

This means IME is fully available. E.g., when an editable element such as <input type="text">, <textarea> or <foo contentediable> has focus.

DISABLED

This means IME is not available. E.g., when a non-editable element has focus or no element has focus, the desired IME state is DISABLED.

PASSWORD

This means IME state should be the same as the state when a native password field has focus. This state is set only when <input type="password"> (ime-mode: auto;), <input type="text" style="ime-mode: disabled;"> or <textarea style="ime-mode: disabled;">.

PLUGIN

This is set only when a windowless plugin has focus.

Be careful, even if a password field has focus, mIMEState may be ENABLED. When you need to check if a password field has focus for security reasons, you should use InputContext::IsPasswordEditor().

The other is IME open state:

DONT_CHANGE_OPEN_STATE

The open state of IME shouldn't be changed. I.e., Gecko should keep the last IME open state.

OPEN

Open IME. This is specified only when ime-mode of the new focused element is active.

CLOSE

Close IME. This is specified only when ime-mode of the new focused element is inactive.

E.g., on Linux, applications cannot manage IME open state. On such platforms, this is ignored.

IME open state should be changed only when nsIWidget::SetInputContext() is called at DOM focus change because changing IME open state while an editor has focus makes users confused. The reason why nsIWidget::SetInputContext() is called is stored in InputContextAction::mCause.

How does Gecko disable IME in IMM mode on Windows

Every window on Windows is associated an IMContext. When Gecko disables IME, mozilla::widget::IMEHandler::SetInputContext() disasociates the context from the window.

How does Gecko disable IME in TSF mode on Windows

mozilla::widget::TSFTextStore sets focus to a dummy context which disables the keyboard.

How does Gecko disable IME on Mac

mozilla::widget::TextInputHandler::HandleKeyDownEvent() doesn't call focused view's interpretKeyEvents. This prevents native key events to be passed to IME.

How does Gecko disable IME on GTK

mozilla::widget::IMContextWrapper sets focus to a dummy context which doesn't have IME composition.

How does Gecko disable IME on Android

?

mHTMLInputType

The value is a string representing the focused editor.

"text", "password", "number", etc.

When an <input> element gets focus, the value is the type of the input element.

"textarea"

When a <textarea> element gets focus, the value is "textarea".

""

When an HTML editor (an element whose contenteditable attribute is "true" or document whose designMode is "on") gets focus, the value is empty. And also, when the other elements get focus.

mHTMLInputInputMode

The value is inputmode attribute value of the focused editor. This is set only when "dom.forms.inputmode" pref is true.

mActionHint

The value is mozactionhint attribute value of the focused editor. This is useful for deciding the caption for the submit button in virtual keyboard. E.g., the value could be "Go", "Next" or "Search".

Native IME handlers

Following classes handles IME on each platform:

Windows

mozilla::widget::IMEHandler

This class manages input mehtod context of each window and makes IMMHandler or TSFTextStore work with active IME and focused editor or windowless plugin. This class has only static members, i.e., never created its instance.

mozilla::widget::IMMHandler

This class is used when TSF mode is disabled by pref ("intl.tsf.enable"), focused content is a windowless plugin or active IME is for IMM (i.e., not TIP for TSF).

This class handles WM_IME_*  messages and uses Imm*() API. This is a singleton class since Gecko supports only on IM context in a process. Typically, a process creates windows with default IM context. Therefore, this design is enough (ideally, an instance should be created per IM context, though). The singleton instance is created when it becomes necessary. So, if user doesn't meet a windowless plugin nor use IME on it, this instance is never created.

mozilla::widget::TSFTextStore

This class handles IME events in TSF mode and when TIP (IME implemented with TSF) is active. This instances are created when an editable element gets focus and released when it loses focus.

TSFTextStore implements some COM interfaces which is necessary to work with TIP. And similarly, there is a singleton class, TSFStaticSink, to observe active TIP changes.

TSF is the most complicated IME API on all platforms, therefore, design of this class is also very complicated.

FIrst, TSF/TIP requests to lock the editor content for querying or modifying the content or selection. However, web standards don't have such mechanim. Therefore, when it's requested, TSFTextStore caches current content and selection with WidgetQueryContentEvent. Then, it uses the cache to reply to query requests, and modifies the cache as they requested. At this time, TSFTextStore saves the requests of modification into the queue called PendingAction. Finally, after unlocking the contents, it flushes the pending actions with dispatches WidgetCompositionEvents via TextEventDispatcher.

Then, IMEContentObserver will notify some changes caused by the dispatched WidgetCompositionEvents (they are notified synchronously in chrome or non-e10s mode, but asynchronously from a remote process in e10s mode). At this time, TSFTextStore may receive notifications which indicates web application changes the content differently from cache in TSFTextStore. However, TSFTextStore ignores such fact temporarily until the composition is finished completely. The reason is that, notifying unexpected text or selection changes to TSF and/or TIP during composition may behave them odd.

When a composition is committed and it receives NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED, TSFTextStore clears the cache of contents and notifying TSF of merged text changes and the last selection change if they are not caused by composition. By this step, TSF and TIP may sync its internal cache with actual contents.

Note that if new composition is started before NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED notification, TSFTextStore handles the new composition with cached contents which may be different from actual contents. So, e.g., reconvertion around caret may not work as unexpectd in such case, but we don't have a good solution for this issue.

On the other hand, TSFTextStore cannot cache character rects since if there are a lot of characters, caching the rects require a lot of CPU cost (to compute each rect) and memory. Therefore, TSFTextStore will use insertion point relative query for them bug 1286157. Then, it can retrieve expected character's rect even if the cache of TSFTextStore is different from the actual contents because TIP typically needs caret position's character rect (for a popup to indicate current input mode or next word suggestion list) or first character rect of the target clause of current composition (for a candidate list window of conversion).

Mac

Both IME and key events are handled in TextInputHandler.mm.

mozilla::widget::TextInputHandlerBase is the most base class. mozilla::widget::PluginTextInputHandler inherits TextInputHandlerBase and handles key events and IME events on focused plugin. mozilla::widget::IMEInputHandler inherits TextInputHandlerBase and handles IME related events. mozilla::widget::TextInputHandler inherits TextInputHandlerBase and implements NSTextInput protocol of Cocoa. The instance is created per nsChildView instance.

GTK

mozilla::widget::IMContextWrapper handles IME. The instance is created per top level window.

Android

nsWindow::GeckoViewSupport handles native IME events and just dispatches Widget*Event.