Commit Graph

2536 Commits

Author SHA1 Message Date
Yohei Yukawa
7a54d14bad Fix potential mozc_renderer launch failures
This CL partially reverts cl/515199465 and e7f823f23f so that fields in `RendererLauncher` can be explicitly initialized.  At least Visual C++ 2022 debug build does require these initializations.

Hopefully having explicit initial values in source code also helps future readers anyway.

Fixes google/mozc#702

PiperOrigin-RevId: 519476998
2023-03-26 07:13:36 +00:00
Yohei Yukawa
d7ac4cc9b9 Run test sequentially in build_mozc.py runtests by default
If the build machine has multiple CPU cores, `build_mozc.py runtests` executs tests in parallel mode by default. However the parallel mode makes it difficult to see what tests are actually failing in continuous test environment.  The worst case scenario is that `build_mozc.py runtests` fails to show actual failures and dashboard continues to look normal.

PiperOrigin-RevId: 519465778
2023-03-26 06:06:58 +00:00
Yohei Yukawa
187c9f7af1 Set PYTHONUTF8=1 for Windows GYP build
Currently there is a chance that Mozc GYP build fails on Windows because some of GYP python files do not explicitly specify file encoding when loading source files thus python uses `locale.getpreferredencoding()`, which can be affected by  `"Language for non-Unicode programs"` in the OS global settings. Note that Mozc's own python files are supposed to be explicitly specifying input file encoding thus this is not the case any more.

In order to mitigate such nondeterministic behaviors at the GYP level, this CL sets `PYTHONUTF8` environment variable for Python 3.7 so chat Python UTF-8 mode (PEP 540) can be explicitly enabled for GYP.

Note that Python is expected to enable UTF-8 mode by default from Python 3.15 (PEP 686). Hopefully opting in that new behavior a bit earlier would make our lives easier.  Once Windows build switch to Bazel, this hack should no longer be necessary anyway.

 - PEP 540: https://peps.python.org/pep-0540/
 - PEP 686: https://peps.python.org/pep-0686/

PiperOrigin-RevId: 519166596
2023-03-24 16:42:04 +00:00
Noriyuki Takahashi
559e8108b4 Replace reinterpret_cast by memcpy.
PiperOrigin-RevId: 519104391
2023-03-24 10:54:47 +00:00
Mozc team
b79bb8578a LSC: C++ DebugString to AbslStringify
In this LSC, we replace C++ protobuf DebugStrings with implicit AbslStringify conversions or absl::StrCat (go/totw/215). This makes debug information incompatible with TextFormat parsers (go/explicit-debug-string).

PiperOrigin-RevId: 518959344
2023-03-23 21:09:11 +00:00
Hiroyuki Komatsu
8598762335 Refactoring of NBestGenerator
* No logic should be changed.

#codehealth

PiperOrigin-RevId: 518832562
2023-03-23 12:25:44 +00:00
Hiroyuki Komatsu
8782b5ae16 Update a comment.
PiperOrigin-RevId: 518741438
2023-03-23 03:06:59 +00:00
Yohei Yukawa
ae14d94980 Deprecate IMM32 stuff from renderer_command.proto
Now that IMMM32 Mozc got removed, we can start deprecating IMM32-only stuff from renderer.

As the first step, this CL clarifies what are used only for IMM32 in `renderer_command.proto` with `[deprecated = true]`.

PiperOrigin-RevId: 518600795
2023-03-22 17:06:36 +00:00
Hiroyuki Komatsu
3c4d4844b1 Update filtering rules for redundant candidates.
* Without this change, candidates are filtered if another candidate has the same key and value.
* With this change, candidates are filtered only if another candidate has the same key, value, lid, and rid.
* This improves word ranking and candidate rewriting.
* Redundant candidates will be deduplicated in another layer.
* https://github.com/google/mozc/issues/689

#typingquality

PiperOrigin-RevId: 518537368
2023-03-22 21:01:58 +09:00
Tomoki Nakagawa
06417c6242 Migrate //engine:engine_builder to mozc::Thread2.
PiperOrigin-RevId: 518514566
2023-03-22 18:39:14 +09:00
Toshiyuki Hanaoka
f7ad867453 Use fallback offset cost for single kanji entries.
I found some negative results without this change:
 Key: Top result value
 ああ: 吁
 あれ: 荒
 あの: 穴
 どこ: 床
 ほご: 夸

Costs for single kanji prediction entries were decided based on other single kanji entries.

When there is no single kanji entries from other sources (UNIGRAM, REALTIME, etc), the offset cost were too small and relatively minor single kanji entries can be ranked high.

PiperOrigin-RevId: 518503396
2023-03-22 17:48:25 +09:00
Mozc team
832d55b5cf Fix 2 ClangTidyReadability findings:
* redundant string conversion

#codehealth

Tested:
    Local presubmit tests passed.
PiperOrigin-RevId: 518495011
2023-03-22 16:51:28 +09:00
Mozc team
344331e6a0 Fix 2 ClangTidyReadability findings:
* redundant string conversion

#codehealth

Tested:
    Local presubmit tests passed.
PiperOrigin-RevId: 518491970
2023-03-22 16:30:47 +09:00
Mozc team
bea782560d Fix 3 ClangTidyReadability findings:
* redundant string conversion

#codehealth

Tested:
    Local presubmit tests passed.
PiperOrigin-RevId: 518486908
2023-03-22 15:57:01 +09:00
Hiroyuki Komatsu
fb007d83dc Refactoring of QualityRegressionUtil
* Added the deduplication feature to GetRank.
  + GetRank skips counting candidates, if the same values have been already checked.
  + This deduplication is also performed in production in another layer.

#codehealth

PiperOrigin-RevId: 518482618
2023-03-22 15:22:18 +09:00
Toshiyuki Hanaoka
4afa27051f Remove expanded chars for the intended modifier key from type corrected queries.
PiperOrigin-RevId: 518460719
2023-03-22 12:59:08 +09:00
Mozc team
8baccbfce4 Fix 2 ClangTidyReadability findings:
* redundant string conversion

#codehealth

Tested:
    Local presubmit tests passed.
PiperOrigin-RevId: 518202052
2023-03-21 17:30:59 +09:00
Mozc team
b20d753656 Fix 2 ClangTidyReadability findings:
* redundant string conversion

#codehealth

Tested:
    Local presubmit tests passed.
PiperOrigin-RevId: 518176322
2023-03-21 14:46:43 +09:00
Yohei Yukawa
d69f5f7afa Include what you use in win32/base/uninstall_helper.h
#codehealth

PiperOrigin-RevId: 518063887
2023-03-21 05:41:56 +09:00
Yohei Yukawa
5fee24fa74 Remove unused stuff from win32/base/uninstall_helper.cc
This CL removes the following unused methods from `win32/base/uninstall_helper.cc`.

- `utow`
- `GetPreloadLayoutsMain`
- `BroadcastNewTIPOnVista`
- `EnableAndSetDefaultIfLayoutIsTIP`

#codehealth

PiperOrigin-RevId: 518055408
2023-03-21 05:11:53 +09:00
Yohei Yukawa
32f19e3fd7 Delete win32/base/immdev.h
`win32/base/immdev.h` is no longer used thus can be removed safely.

#codehealth

PiperOrigin-RevId: 518048206
2023-03-21 04:44:43 +09:00
Yohei Yukawa
b8e63d231b Delete unused ImmRegistrar
`ImmRegistrar` is no longer used thus can be removed safely.

#codehealth

PiperOrigin-RevId: 518016952
2023-03-21 02:55:35 +09:00
Mozc team
d89e8dcefc LSC: Sort additional lists with strings and labels
The change introduces sorting of list values of new rule attributes:
  * test_data
  * test_deps
  * test_srcs
  * test_tags
  * private_deps

The values of the attributes mentioned above as well as all other attributes
that required sorting before (such as `srcs`, `deps`, `data`, etc) will now
also be sorted if the values are not single lists but concatenations of lists
and/or select statements.

More information: go/lsc-sort-labels

Tested:
    TAP train for global presubmit queue

PiperOrigin-RevId: 517918293
2023-03-20 19:20:50 +09:00
Yohei Yukawa
1aa495b283 Assume SystemUtil::IsWindows8OrLater() is always true in imm_util.cc
#codehealth

PiperOrigin-RevId: 517428981
2023-03-18 01:14:13 +09:00
Yohei Yukawa
8b0f347260 Assume SystemUtil::IsWindows8OrLater() is always true in migration_util.cc
#codehealth

PiperOrigin-RevId: 517423412
2023-03-18 00:49:04 +09:00
Toshiyuki Hanaoka
b29e6f76a2 Remove noisy single kanji entries
PiperOrigin-RevId: 517384650
2023-03-17 20:57:59 +09:00
Hiroyuki Komatsu
cf3e496e37 Add 一本(いっぽん) of the same POS IDs with 一個(いっこ).
* https://github.com/google/mozc/issues/689
* This does not solve the issue yet, because another entry of 一本 with different POS IDs is prioritized.

#typingquality

PiperOrigin-RevId: 517307936
2023-03-17 13:28:57 +09:00
Noriyuki Takahashi
a76205ee7e Accept absl::string_view for Mmap::Open().
PiperOrigin-RevId: 517028944
2023-03-16 14:55:01 +09:00
Toshiyuki Hanaoka
ba6c4aaba6 - Make ResultCostLess() consistent with the function name.
- Add length comparison for value string.
  - The default operator `<` does not compare strings in UTF-8.

PiperOrigin-RevId: 516985769
2023-03-16 10:36:07 +09:00
Yohei Yukawa
76ec18450e Remove unused MigrationUtil::RestorePreload().
This method is no longer used thus can be removed safely.

#codehealth

PiperOrigin-RevId: 516955761
2023-03-16 08:07:38 +09:00
Yohei Yukawa
8f875f2df8 Remove unused ImeUtil::IsDefault().
This method is no longer used thus can be removed safely.

#codehealth

PiperOrigin-RevId: 516861656
2023-03-16 02:21:09 +09:00
Yohei Yukawa
e7c4247ece Delete unused FocusHierarchyObserver and its related classes.
With my previous CL the following classes are no longer used anywhere.  Let's remove them for better code health.

- `AccessibleObject`
- `AccessibleObjectInfo`
- `BrowserInfo`
- `FocusHierarchyObserver`

#codehealth

PiperOrigin-RevId: 516844278
2023-03-16 01:17:36 +09:00
Tomoki Nakagawa
c51876ab82 Refactor and fix concurrency issues in //renderer::renderer_client.
- Add thread annotations.
- Based on the comment "// |renderer_status_| is also protected by mutex.", mark it as `ABSL_GUARDED_BY` too
- Fix concurrency issues discovered by the added thread annotations.
- Use `enum class` instead of `enum`.
- Use `std::optional<T>` instead of `std::unique_ptr<T>` since the usage was just for nullability.
- Migrate to `mozc::Thread2`.
- include-what-you-use.

PiperOrigin-RevId: 516766669
2023-03-15 18:29:47 +09:00
Yohei Yukawa
4e237315e4 Remove dependency on FocusHierarchyObserver from win32/tip/
As the first step towards deprecating `FocusHierarchyObserver`, this CL completely removes the dependency on it and `BrowserInfo` from files under `win32/tip/`.

Note that there must be no user-visible impact as the feature has been disabled since cl/102450708.

#codehealth

PiperOrigin-RevId: 516680258
2023-03-15 09:43:58 +09:00
Yohei Yukawa
c777afc872 Stop supporting IMM32 Mozc in UninstallHelper::GetNewEnabledProfileForVista().
We no longer need to consider the scenario for Windows XP/Vista/7 where IMM32 Mozc is installed on the machine.  This means that we can safely simplify `UninstallHelper` class.

This CL removes such a legacy handling code for Windows 7.

This CL also removes `UninstallHelperTest::BasicCaseForVista()` as it is not a supported scenario any more.

#codehealth

PiperOrigin-RevId: 516672000
2023-03-15 09:00:56 +09:00
Toshiyuki Hanaoka
97ad827d99 Bug fix: Check seen_ map before checking other conditions not to increment counters.
PiperOrigin-RevId: 516488710
2023-03-14 20:42:08 +09:00
Yohei Yukawa
94b6355989 Remove UninstallHelper::RemoveHotKeyForVista().
This method has been used to clean up any user-defined hotkey associated with IMM32 Mozc, which should no longer exist.  Not calling this method should have no user impact.

#codehealth

PiperOrigin-RevId: 516432892
2023-03-14 15:16:36 +09:00
Yohei Yukawa
101700df11 Amend UninstallHelperTest::BasicCaseForWin8
For some reasons `UninstallHelperTest::BasicCaseForWin8()` has been testing an unusual scenario where both IMM32 Mozc and TSF Mozc are installed at the same time.

This CL updates the test scenario for the real scenario.

PiperOrigin-RevId: 516429432
2023-03-14 14:54:32 +09:00
Yohei Yukawa
795bcd5792 Delete DisableLegacyMozcForCurrentUserOnWin8()
This is a logical rollback of changelist 39183361.

#### Reason for rollback

`MigrationUtil::DisableLegacyMozcForCurrentUserOnWin8()` was added more than 10 years ago as a quick workaround for users who upgraded from Windows 7 or prior OSes.  Let's roll it back as we no longer support such upgrading scenario.

#### Original change description

Add a quick workaround to unregister IMM32 Mozc on Windows 8.

---

PiperOrigin-RevId: 516419611
2023-03-14 14:05:16 +09:00
Yohei Yukawa
d097ec8c49 Assume SystemUtil::IsWindows8OrLater() is always true in ime_switcher.cc
#codehealth

PiperOrigin-RevId: 516390814
2023-03-14 11:28:50 +09:00
Yohei Yukawa
e0bbc03371 Assume SystemUtil::IsWindows8OrLater() is always true in prelauncher.cc
#codehealth

PiperOrigin-RevId: 516386620
2023-03-14 11:00:50 +09:00
Hiroyuki Komatsu
2340b22b37 Skip filtering candidates of user dictionary.
If the candidate comes from the user dictionary, it is not filtered regardless the capability of rendering.

PiperOrigin-RevId: 516372972
2023-03-14 09:51:46 +09:00
Yohei Yukawa
814606b5fe Let Mozc binaries require Windows 8 or later.
Just to be consistent with recent WIX rule changes, with this CL all Mozc binaries start requring Windows 8 and later.

See cl/114053403 for the previous version bump.

Note that we'll most likely start requiring Windows 10 and later soon.

PiperOrigin-RevId: 516348145
2023-03-14 08:01:30 +09:00
Emma Haruka Iwao
8b69a84dbc Implement PlatformString and PlatformStringView
PlatformString is an alias to std::basic_string, and PlatformStringView is a null-terminated platform-native string view. These classes aim to address a few pain points:

- absl::string_view doesn't guarantee null-termination and cannot be used for C-style APIs.
- Sometimes multiple conversions happen when calling APIs on Windows.
- consolidate platform conditionals into one build rule

<path/to/mozc>/base/file_stream.cc has ToPlatformString and this is a more complete version of that implementation.

PlatformStringView implements most of the C++17 string_view member functions, but lacks some operations like find, find_first_of, etc because the focus of the class is integration with C APIs. We can add them in the future if deemed necessary.

PiperOrigin-RevId: 516336863
2023-03-14 07:18:15 +09:00
Emma Haruka Iwao
82beec68ca Remove OutputUtil
It was used by the IMM32 implementation and is no longer needed.

#codehealth

PiperOrigin-RevId: 516323901
2023-03-14 06:30:30 +09:00
Toshiyuki Hanaoka
c9c94eb423 Add a experimental feature for the new Typing Correction Scoring.
PiperOrigin-RevId: 516093274
2023-03-13 12:13:13 +09:00
Hiroyuki Komatsu
44da945538 Fix a build error of symbol_rewriter.cc
* Added absl/strings/str_cat.h for absl::StrAppend.
* Follow-up to cl/515095644.

#codehealth

PiperOrigin-RevId: 516078918
2023-03-13 10:15:31 +09:00
Emma Haruka Iwao
7a915faa39 multifile: Pass the string to InputMultiFile
InputMultiFile takes a string_view, so instead of passing a c_str, pass it directly.

Remove the explicit ifs_.reset call in the destructor because the unique_ptr destructor handles it.

PiperOrigin-RevId: 516073040
2023-03-13 09:18:35 +09:00
Emma Haruka Iwao
a36c42a340 Add a POSIX fallback implementation for DeleteRecursively.
PiperOrigin-RevId: 516071131
2023-03-13 09:00:11 +09:00
Emma Haruka Iwao
da6c973aae Make TempDirectory/TempFile move constructors noexcept
STL contaieners don't use them if they're not noexcept. std::string's move constructor and swap are noexcept, so we can mark them noexcept.

PiperOrigin-RevId: 516070982
2023-03-13 08:58:40 +09:00