opral / inlang-message-sdk Goto Github PK

Context

Place to discuss everything about the Import/Export project

Context

Messages are currently stored in a solidjs reactive map which consumes excessive resources and does not scale well.

Replacing the internal solidjs reactive map for lint-reports successfully showed that this approach makes sense, as long as there is another notification mechanism for apps to react to changes.

Proposal

• replace the internal reactiveMap with a vanilla js Map
• trigger message update events independently when updates are handled (see loadProject.ts)
• [tbd] Find a way to use the solidAdapter to make messages which are in-view reactive at the app-level.

testing please ignore

This is a suggestion for how to make the bug go away

• do this
• then do that
• #5
• opral/monorepo#1772
• opral/monorepo#2162

Context

Proposal

Context/Problem

project.query.message.getAll() sometimes returns undefined instead of a message array. This isn't reflected in the types.

The most likely cause is that the messages get queried before they are properly loaded & thus the query returns an uninitialized value.

This caused PARJS-57

Proposal

Either block until the messages are fully loaded before responding to the query, or return an empty array.

MESDK-59

OG issue & discussion: opral/monorepo#1892

The current experimental implementation of aliases is incomplete
Only the default alias is supported for plugin message IDs, and uniqueness is assumed.

Next steps

• New message creation with auto-generated human id (no alias)
• Improve Sherlock and Fink messages UI (e.g. "@" for aliases feels rough)
• Improve tooling to connect messages wtih context / code
• (?) additional alias types e.g. for mobile frameworks

Additional TODOs migrated from opral/monorepo#2346

• review docs
• how to handle (or disallow) alias non-uniqueness
• how to handle (or disallow) alias updates
• review Fink search by alias https://github.com/opral/monorepo/pull/2108/files#r1510753012

Discussed in opral/monorepo#2126

• MESDK-50
• Continuation of MESDK-6

Context/Problem

The current sdk implementation does not scale well. Reactive solid.js effects for messages and lint reports consume excessive CPU and memory resources as the number of messages grows. (see MESDK-29 and loom)

Proposal

• Remove solid.js from the sdk core
• core should offer a CRUD api for projects, messages + watcher api for events
• offer a separate reactive api for building UIs.

from https://app.excalidraw.com/s/1RmnkzJA3Ph/27CVKAmMGKx

Context

Proposal

Context

E.g. The type declaration requires that modules URLs end in `.js`, but the error message when this is not the case, is unclear:

ProjectSettingsInvalidError: The project settings are invalid:

"/modules/0":

Expected all values to match
    at parseSettings (file:///Users/jldec/opral/monorepo/inlang/source-code/cli/dist/main.js:88702:13)
    at loadSettings (file:///Users/jldec/opral/monorepo/inlang/source-code/cli/dist/main.js:88695:10)

Proposal

Either improve the error, or remove the check if the error can't easily be improved
In this case, most bad urls will be caught by a module load error or another runtime error.

Partial support for logging was implemented with the debug package e.g. in sdk/load-test.

• make it work in browsers
• extend support to other parts of the sdk using the sdk:module convention e.g. sdk:lintReports

Context

sdk-persistence means that we can improve how the inlang sdk handles upgrades as we add features.

There is currently no migration mechanism for plugins. This means that users can (and do) get stuck on outdated plugins.

(correction April 23. The inlang message format plugin does have a migration capability for v1.x -> v2.x, documented in the changelog.)

Proposal

1. Existing projects using plugins should be able to migrate to sdk persistence
   • Initally this can be a one-time, user-initiated process, e.g. triggered by the inlang CLI
   • Once stable, we should nudge users and offer to do this also from other apps
2. The sdk should detect internal persisted format changes, and auto-migrate the sdk message store, without user intervention.
   • Some backward and forward sdk <-> file-format compatiblity is needed (details tbd)
   • Ensure all-or-nothing (atomic) migrations, and avoid file corruption.

Context

We define interfaces of the inlang sdk in dedicated packages. The idea was to decrease the impact of breaking changes of interfaces on the ecosystem. However:

• plugins and lint rules import the sdk regardless of dedicated interface packages
• more complex codebase
• iteration speed slightly impacted see discussions like opral/monorepo#2700 (comment)

Proposal

Remove the versioned interfaces and put them in the SDK.

Video

https://www.loom.com/share/ca91189f093940c38ae0f2927725b06c?sid=93b4ddf6-ce84-4581-91f3-97987224652d

https://discord.com/channels/897438559458430986/1218873582726156338

https://github.com/mishodzuliashvili/one-punch-man

to reproduce

git clone https://github.com/mishodzuliashvili/one-punch-man.git
cd one-punch-man
rm project.inlang/project_id
code .

MESDK-60
Migrated from opral/monorepo#2333

Proposal

User reporting

error when starting dev server:
ReferenceError: structuredClone is not defined
@inlang/sdk/dist/createMessagesQuery.js:18:31

The sdk error should report node version and ask user to upgrade (for cli and paraglide)

see https://discord.com/channels/897438559458430986/1220565262320009317/1222471930297323550

Context

Proposal

testing please ignore

problem

• MESDK-11

• MESDK-45

• opral/monorepo#1889 (comment)

• #10

  opral/monorepo#1889 (comment)

fix

• identify the problem
• fixit

I want to build the settings ui component based on the schema I get from the installed lint rules and plugins. The installed API only gives me access to id displayName description module.

Porposal:

• Add settingsSchema to types:

export type InstalledPlugin = {
	id: Plugin["id"]
	displayName: Plugin["displayName"]
	description: Plugin["description"]
	/**
	 * The module which the plugin is installed from.
	 */
	module: string
	// disabled: boolean
+	settingsSchema: Plugin["settingsSchema"]
}

Same for message lint rule...

• Add settingsSchema to InstalledPlugin & InstalledMessageLintRule:

        const installedPlugins = () => {
			if (!resolvedModules()) return []
			return resolvedModules()!.plugins.map((plugin) => ({
				id: plugin.id,
				displayName: plugin.displayName,
				description: plugin.description,
				module:
					resolvedModules()?.meta.find((m) => m.id.includes(plugin.id))?.module ??
					"Unknown module. You stumbled on a bug in inlang's source code. Please open an issue.",
+                settingsSchema: plugin.settingsSchema,
			})) satisfies Array<InstalledPlugin>
		}

Same for message lint rule...

It seems like an easy PR. @jurgen.leschner, let me know if I should open a PR to add this.

Problem

Lint rules are all or nothing. There is no way to ignore a warning coming from a lint rule just for a specific message.

This is especially annoying with the "duplicate pattern" lint rule which has a lot of false positives.

This has been a common complaint

• https://discord.com/channels/897438559458430986/1220012877768032389
• opral/monorepo#2408

Proposal

Some way to ignore lint warnings for specific messages.

This would likely require annotating messages with metadata.

cc @felix.haeberle

MESDK-70

Context

https://github.com/opral/monorepo/pull/2590/files#discussion_r1567809135

"pathPattern": "./messages/{languageTag}.json", results in a tree like this:

.
├── messages
│   ├── de.json
│   └── en.json
├── project.inlang
│   └── settings.json

That puts the files in a peer directory called messages, not inside the project directory (i think this is from the time when project settings were just a json file.)

Proposal

We'll need to define a new way to specify paths inside the project, without assuming a specific project {name}.inlang in the path

Context/Problem

1. sometimes forks/clones are actually new repos and should have a new projectid
2. some users would prefer not to send telemetry with a projectid

Proposal

1. offer an easy way to regenerate the projectid
2. option to show or disable telemetry - see https://nextjs.org/telemetry

cc: @samuel.stroschein

will the comments sync into GitHub?

I wish auto-sync'ed Linear issues would include a link back to Linear
MESDK-49

MESDK-64

Context

There is currently no storage plugin which supports the full inlang Message structure.

This is required in order to unblock features like plurals and gender e.g. in paraglide.

Proposal

• use the existing plugin API for loadMessages and saveMessages (no rewrite of loadProject)
• store under project directory {name}.inlang/messages/…
• enable with experimental persistence feature flag in project settings
• NOTE: Non-human-editable

Context

This is a discussion issue for sdk persistency project requirements.

Why do we need persistence?

Inlang projects should provide built-in persistence instead of relying on developers to create text files with formats which may not support globalization needs.

Our goals with this work

1. To simplify the inlang getting-started experience.
   Users create new inlang projects in 1 click => A project has built-in message storage.
2. To support high-demand features like plurals, gender etc.
   Inlang projects store message variants, match patterns etc.
3. To offer a smooth transition from directly editing messages in text files,
   to editing messages via a UI like Fink or Sherlock.
4. To enable change-driven workflows e.g. across translators, developers, and designers.
   This means (a) mapping persisted message state changes to repo diffs/merges/commits, and (b) persisting repo workflow state like reviews and approvals alongside the persisted messages.
5. To enable high quality, context aware, AI-assisted globalization.

Timeline considerations

In the short term, feature coverage (plurals, gender etc.) and human editability are most important. This speaks for shipping MVP persistence together with editor support ASAP.

Context

This discussion came up while using the name of an inlang project in Sherlock. It would be great to have a project property called name which is the name of the inlang project.

Proposal

The name of the inlang project is most likely just the directory (file) name of the inlang project itself such as project.inlang, backend.inlang or happy-elephant.inlang

----

@samuel.stroschein said in SHERL-22:

EDIT: Is there a way to get the project name from the SDK API? Or can we only get it by reading the file path?

Only the file path name.

Suggestion: Pink message sdk team if adding project.name makes sense.

Proposal

Remove the tryCatch stuff from message SDK in favor of standard JS.

Value of tryCatch is close to 0

@samuel.stroschein said in LIX-59:

EDIT: never mind. the value of tryCatch is close to 0.

@jldec what is your issue with the utility function tryCatch()? The function doesn't impact API design.

const file = await tryCatch(() => repo.fs.readFile())

if (file.error){
  // do something
}

const parsed = tryCatch(() => JSON.parse(file.value))

if (parsed.error){
  // do something
}

try {

  const file = await repo.fs.readFile()
  const parsed = JSON.parse(file)

} catch (e){
  if (e instanceof FileError){
    //
  } else if (e instanceof JSONError){
    //
  }
}

Context

Proposal

Context

Users are asking for inlang to support messages with variant for gender and plurals.

Proposal

Introduce new types which are aligned with ICU Message Format 2 messages and a new message bundle type to collect messages translated for different locales.

Context

linear projects do not support comments or discussions
but issues do.

Proposal

Context

testing

Proposal

Context

While pulling plugin modules e.g. for lint reports from a cdn is useful in environments which don't have a package manager (e.g. in browsers), the current behavior is surprising in other environments where installed packages are supported (e.g. in locally installed npm projects or in a vs code extension)

Please use this issue to report requirements.

• performance: network requests are expensive e.g. on each paraglide compile
• offline usage: could be mitigated by preloading and caching
• security checks: would be easier with a trusted/enterprise package registry
• plugin user DX: configuring and knowing which plugin is going to run should be simple
• plugin developer DX: how to dev-test-build-ship new plugins and plugin-updates should be straightforward
• reliability: depending on a 3rd party cdn which is not typically used for app hosting, adds operational risk
• observability: it should be possible to trace plugin calls to debug when something is not right

Trying to understand how comments sync with Linear.
MESDK-48

Context

Proposal

Discussed in opral/monorepo#2253

// unlimited inboxes
<p>{m.elephant_big_tree()}</p>

// manage team members
// 💥 error m.elephant is used multiple times
<p>{m.elephant_big_tree()}</p>

// spam report
// 💥 error m.elephant is used multiple times
<p>{m.blue_box({ title: m.elephant_big_tree() })}</p>

Context

https://github.com/opral/monorepo/blob/main/inlang/source-code/sdk/src/createNewProject.ts#L13

Proposal

@samuel.stroschein said in SHERL-3:

@felixhaeberle you want listProjects.subscribe()?

Currently if a loading-error happens in a Plugin during message loading, perhaps due to a syntax error, it's quietly ignored. We should add a way to report loading issues from plugins so we can inform developers why loading is not working

Context

The current MessageQueryApi operates on Message objects.

export type MessageQueryApi = {
	create: (args: { data: Message }) => boolean
	get: ((args: { where: { id: Message["id"] } }) => Readonly<Message>) & {
		subscribe: (
			args: { where: { id: Message["id"] } },
			callback: (message: Message) => void
		) => void
	}
	// use getByDefaultAlias() to resolve a message from its alias, not subscribable
	getByDefaultAlias: ((alias: Message["alias"]["default"]) => Readonly<Message>) & {
		subscribe: (alias: Message["alias"]["default"], callback: (message: Message) => void) => void
	}
	includedMessageIds: Subscribable<Message["id"][]>
	/*
	 * getAll is deprecated do not use it
	 */
	getAll: Subscribable<Readonly<Message[]>>
	update: (args: { where: { id: Message["id"] }; data: Partial<Message> }) => boolean
	upsert: (args: { where: { id: Message["id"] }; data: Message }) => void
	delete: (args: { where: { id: Message["id"] } }) => boolean
}

☝️ is exposed as project.query.messages in the InlangProject api

export type InlangProject = {
//  ....
	query: {
		messages: MessageQueryApi
		messageLintReports: MessageLintReportsQueryApi
	}
}

Proposal

From MESDK-77

after introducing UUIDs for variants / we will have to expose a project.query.variant.* next to project.query.messageBundle.* and project.query.message.*.

The comment above suggests a new MessageBundleQueryApi for apps to operate directly on the MessageBundle structure.

@samuel.stroschein @martin.lysk1
Given our experience with MessageQueryApi, do you think this proposal the right approach to use at this time?

MESDK-47

Context

1. The test will always touch the checked in file project4-dir/project.inlang/messages.json which messes with nx test cache. (i think)
2. The same file is sensitive to being manually saved with whitespace changes (after prettier formatting), making it show up unexpectedly in PRs like here

Proposal

Fix the test not to require a checked in json file with "abnormal" whitespace.

Discussed in opral/monorepo#1961

MESDK-71

Context

Any modification of project settings should trigger a reload of messages e.g. to pick up new languages.

e.g. https://discord.com/channels/897438559458430986/1070750156644962434/1229849444728442964

Proposal

Doing this internally, at the sdk level, should allow sherlock users to edit project settings directly, and see the results without reloading manually.

NOTE: guard against a new message file overwriting an existing file which existed before the language was added to the project.

related to FINK-27 (opral/inlang-fink#17)

Context

 ❯ src/functions/machineTranslateMessage.test.ts  (5 tests | 1 failed | 1 skipped) 805ms
   ❯ src/functions/machineTranslateMessage.test.ts > should not return escaped quotation marks
     → expected { id: 'mockMessage', alias: {}, …(2) } to deeply equal { id: 'mockMessage', alias: {}, …(2) }

⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯

 FAIL  src/functions/machineTranslateMessage.test.ts > should not return escaped quotation marks
AssertionError: expected { id: 'mockMessage', alias: {}, …(2) } to deeply equal { id: 'mockMessage', alias: {}, …(2) }

- Expected
+ Received

  Object {
    "alias": Object {},
    "id": "mockMessage",
    "selectors": Array [],
    "variants": Array [
      Object {
        "languageTag": "en",
        "match": Array [],
        "pattern": Array [
          Object {
            "type": "Text",
            "value": "'",
          },
          Object {
            "name": "id",
            "type": "VariableReference",
          },
          Object {
            "type": "Text",
            "value": "' added a new todo",
          },
        ],
      },
      Object {
        "languageTag": "de",
        "match": Array [],
        "pattern": Array [
          Object {
            "type": "Text",
            "value": "' ",
          },
          Object {
            "name": "id",
            "type": "VariableReference",
          },
          Object {
            "type": "Text",
-           "value": " ' hat eine neue Aufgabe hinzugefügt",
+           "value": " ' hat ein neues To-Do hinzugefügt",
          },
        ],
      },
    ],
  }

Proposal

fix it.