Once #39 has been merged. We will need to update the web front end to use these new fields instead of the older data structure

Description

The content queries are returning draft documents. They should only return published documents.

Acceptance criteria

• The content queries only return published documents.

Move all custom work into an actual sanity plugin. To do this well need to use Sanity's plugin kit.

• https://github.com/sanity-io/plugin-kit

Description

Once we have the basic building blocks in place, it would be good if we were able to package this up into a Sanity plugin.

Acceptance criteria

• Investigate the viability of wrapping the MVP functionality into a Sanity plugin.
• Present findings to team in a scheduled meeting.

• Transfer ownership to Formidable Repo
• Deploy Web
• Deploy Studio

Depends on #9

Description

This is a follow-on from #9, which implemented on-demand translation of content. This ticket did not account for scenarios where the base content is updated after the translation has been created.

Update the function to check for two conditions: first, the existence of a translation, and second, whether the base content has been updated since that translation was created. If the base content is newer, a new translation should be requested.

Acceptance criteria

• If the base content has been updated since the translation was created, a new translation should be requested and patched/returned accordingly.
• If the base content has not been updated since the translation was created, the existing translation should be returned as-is.

As were coming close to the end of this project, it would be good to think of ways we can share this experiment to the world. We agree the most sensible approach would be creating a blog post.

Feel free to use this ticket as brain dump of topics to cover.

Description

We need a utility function that can be used to translate content from Sanity. The function should take a Sanity document (or JSON string), and an IETF language tag (e.g. es-US), and return a translated version of it.

Acceptance criteria

• The function should use OpenAI's Node API Library for translation.
• The function should be able to translate a Sanity document (in the form of a JSON string) into a specified language.
• The function should be unit-tested. Aborted due to mocking issues with Bun.

Description

To allow us to create a custom plugin to ship it to the world we need to remove the dependancy of the sanity document translations plugin.

Removing this plugin will have the effects mentioned here #32

Acceptance criteria

• Create a localeText custom input
• Create a localePortableText custom input
• Remove the usage of the sanity document translations plugin
• Use the supported languages to provide the backbone of the field structure

Description

We need our Sanity schema to be structured to facilitate the storage of translations for text content.

Update each of the existing document schemas to hold fields for en-US (required), and mi-NZ (optional). It may be useful to consider using the following Sanity plugin: https://www.sanity.io/plugins/i18n-fields.

Acceptance criteria

• Each document schema that displays content on the frontend has separate text fields for en-US (required), and mi-NZ (optional).

Description

Currently the translation functionality is reliant on the output of the LLM being in a specific format. We want to move away from this and instead use a structured output format that can easily be parsed.

Investigate usage of the following pattern for structured output: https://js.langchain.com/docs/modules/chains/popular/structured_output

If this is suitable, then update the translate function to use this format.

Acceptance criteria

• translate function is updated to use structured output
• extractJSON function is removed or updated as required

Can be picked up after #9.

Description

Our current implementation of translations is based on locales. For the purpose of reaching our MVP, we should strip back to languages instead of locales. This will allow us to focus on the core functionality of translations and not worry about the complexity of locales.

Completion of this ticket will require updating the following:

• Sanity Studio code
• Server routing
• Translation API

Acceptance criteria

• if locale includes en we will group all these into "English"
• else store as individual languages

Depends on #7

Description

It would be useful to have nice pre-written content so we can show off the features of the app. This content should be in a separate dataset so that it can be maintained separately from the development dataset.

Acceptance criteria

• A new dataset is created with the name staging.
• The dataset contains life-like content that can be used to demonstrate the app.

• As a user i would like the ability to switch between supported languages at document level. This would filter out localeFields based on the currently selected langauge

Description

Let's stay on the Bun run and use ElysiaJS for our server. It has some nice error handling and should be lightning fast.

Also, it's about time we start dynamically returning content in the user's language rather than just returning the default translation.

Acceptance criteria

• Swap out the existing Express.js server for ElysiaJS
• Use the request headers to determine which language to return
• Sit the API behind a /api route

At the moment we can detect the default language from the schema in sanity.config.ts but ideally we also need to be able to do this within the custom desk structure where we currently have:

const defaultLanguage = {
  id: 'en-US', title: 'English (US)', default: true
};

This could possibly cause some issues if the default language were to ever change in the schema

Acceptance criteria

• The default language should be retreived by the default langauge set in schema

Description

Since we need supported languages to be defined for Sanity to structure the references between documents, since we cannot hard code this, we need a schema that we can query any documents (languages) stored there to use.

Acceptance criteria

• There is a supported languages schema

In our recent frontend (FE) refactor, we eliminated the API that was powered by Elysia JS, a web framework enhanced by Bun. While this API was a perfect fit for our headless architecture, further investigation suggests that Fastify might be a superior choice for our needs. The primary advantage of Fastify lies in its reliance on the Node.js runtime, which enjoys broader support and compatibility compared to the Bun runtime. This change could significantly expand our deployment options and potentially offer other benefits like improved performance, better community support, and advanced features.

Required Routes

• GET - /api/documents/:type?dataset="DATASET"&projectId="PROJECT_ID"
• POST - /api/translate?dataset="DATASET">&projectId="PROJECT_ID"

Optional Routes

• GET - /posts

Description

Now that we have a REST API that queries content from Sanity, we can start using it in the frontend.

Update the frontend to fetch content from the server and display it. Don't worry much about styling as this is just a POC for now, but feel free to use Tailwind classes for some basic styles.

Acceptance criteria

• Frontend is updated to fetch content from Sanity
  • API requests are separated into an /api folder
• Frontend is updated to display content from Sanity
  • React components are separated into a /components folder

Description

This feature will allow content managers to manually trigger the AI-based translation of a document. This will be useful if a content manager wants to shortcut the automatic translation process to ensure that a document is translated as soon as possible.

Acceptance criteria

• A new button is added to the document page that allows content managers to manually trigger the translation of the document.
• When the button is clicked, the document is sent to the translation service and the corresponding translation is added.
• The button is disabled while the document is being translated.
• The button is only visible if the document is not already translated.
• (optional) Multiple translations can be requested at the same time.

Description

This could use some pizzazz:

Acceptance criteria

• Representative icons for each document type

Proposal for Converting to a Monorepo

Overview

We propose converting our current repository, sanity-translations-experiment, into a monorepo. This change aims to streamline our development process and improve project management.

Benefits

1. Better Organization

   • Manage multiple projects in one repository for improved interaction and visibility.

2. Unified Tools and Configurations

   • Standardize tools and configurations across all projects to streamline development.

3. Easier Collaboration

   • Simplify code sharing across projects, enhancing teamwork and efficiency.

4. Streamlined Dependency Management

   • Manage a single set of dependencies for all projects, reducing conflicts and simplifying updates.

5. Efficient Builds with Turborepo

   • Utilize Turborepo for optimized development cycles, with intelligent rebuilding and retesting.

6. Scalable Structure

   • Accommodate growth and new additions without increasing complexity.

7. Simplified Maintenance

   • Centralize maintenance and refactoring efforts, especially for interconnected projects.

8. Consistent CI/CD

   • Implement a unified CI/CD pipeline for consistent deployment processes.

9. Clearer Project Overview

   • Gain a comprehensive view of all projects, enhancing change tracking and historical understanding.

• Give some user feedback while were translating the document (toasts, prompts)

Currently, we have a global README.md file which covers the project at a high level. We need to have or create module level markdown files to explain how each portion works.

Description

Arguably the most important feature of this project is the translation of Sanity content on-demand. We need to modify the getPost function so that, if the requested post is not in the user's language, it will be translated into the user's language before being returned.

Critically, it will also need to be patched back into Sanity so that the next time it is requested, it will be returned in the user's language without needing to be translated again.

Acceptance criteria

• If a translation already exists for the requested post, it should be returned.
• If a translation does not exist for the requested post, it should be translated, patched back to Sanity, and then returned.

Note: For now, we do not need to worry about whether the translations are up-to-date. We will deal with that in a later issue (#10).

Description

Update the frontend to gracefully handle requests which take a while to load. Currently, our server is making a call to Open AI, this can take some time leading to the user sitting around bored. With react-router-dom we can optimistically render the UI so page transitions are almost instant while we wait for content to load

Acceptance criteria

• If a user clicks on a post, the page should change and we should render a skeleton ui while we wait for content to load

Description

This is a follow-on from #3 which had some issues mocking the openai module due to limitations still present in bun: oven-sh/bun#5394.

It may be preferable to just use jest for our tests instead of the bun test runner.

This issue is to finish tests for the translate function.

Acceptance criteria

• Unit tests across the translate function

Missing areas in the README.md

• Plugin
• API (Not yet created)

Description

There is an inconsistent experience between viewing a post, and viewing the list of posts. Individual posts are translated, but the list of posts is not.

Let's update the server to translate the list of posts too.

Note: We may require a new prompt for this. Consider various options and report back to the team before proceeding with the implementation.

Acceptance criteria

• When viewing the list of posts, the titles should be translated.

Description

We need a single source of truth for our linting and prettier config. This will allow us to have a consistent code style across the project, and eliminate noise in PRs.

Acceptance criteria

• An applicable workspaces config in package.json
  • (optional, if desired) The frontend being shifted into a web workspace
• A single source of truth for linting and prettier config
• The following packages are added and configured (why?):
  • eslint-config-prettier
  • eslint-plugin-prettier
• (optional, if desired) The following ESLint rules:
  • import/order
  • import/no-extraneous-dependencies
  • Disabling @typescript-eslint/no-explicit-any
• Linting/formatting run across all workspaces

Description

We need to configure a server-side client for Sanity. This client will be used to make read and write requests to the Sanity datasets.

Acceptance criteria

• A server-side client should be configured for Sanity.
• The client should be able to make read and write requests to the Sanity datasets.

The plugin should come with some basic configuration options. As our plugin is agnostic to the type of schema it should work with. We need to allow users to customise the plugin to work with their data model. The following configuration options would be needed in order for us to provider a "headless" solution.

Required Configurations
If any required options are missed. the plugin should throw an error

• apiKey - Open AI api key
• sanityToken - user generated sanity token, this is needed in order for us to run mutations against their dataset and studio
• schemaTypes - an array of schema types that plugin should render on, a minimum of one schema type must be added

Optional Configurations

• sanityApiVersion - specific api version of sanity the user would like to use. This will default to the latest version if nothing is set
• defaultLanguage - default language/locale that is selected in the custom dropdown on a given document. This will default to an empty string if nothing is set

Description to be completed

Needs #9 to be implemented.

Description

Once The translation work around creating new documents is done, it would be good if we had a caching system in place to return translated documents quickly

Acceptance criteria

• [ ]

Description

We need Sanity to control the content of this app. This ticket is for the initial setup of Sanity.

Acceptance criteria

• Sanity studio is configured and functional

Currently, were sending the entire document to open AI for translating. With the new schema, we will only need to send over the title and content fields. Once #39 has been merged, we will need to update the server code to do this

Acceptance criteria

• Once a translation has been submitted, the corresponding Locale fields should update

Description

Sanity's documentation cautions against using a dynamic date for the apiVersion, which we are doing here, here, and here.

Let's hardcode this to today's date ('today' being the day this ticket gets picked up 😆).

Acceptance criteria

• apiVersion is hardcoded to today's date
• apiVersion is defined in a single place that can be shared between the studio and the server

Description

Now that our app is starting to take shape, we should ensure that the README reflects the ongoing progress and provides clarity about the current implementation details.

Acceptance criteria

• Add a few sentences at the start to describe the app's purpose.
• Add a new section titled 'Implementation Details'
  • Document the current structure and functionality of the app.
• Tick off completed user stories in the 'MVP' section (if any).
• Add a new section titled 'Getting Started' to the README.
  • Document how to run the app locally (mention the chimera of pnpm and bun)
• (optional) Add separate README files for /studio and /server