Logen — Adding your own converter

Hey guys, welcome back! This is my fourth and final article in this series about my cross platform localization generation tool. In this part I will show you an example of how to extend the project to support any localization format.

If you wonder why and how to use my tool, check out part one. Part two provided more information about how it converts one localization format to another one and part three explained the structure of the project in detail.

But now let’s get started!

Let’s say, you have an app that is already localized in English, German and Spanish, but a few users complain about mistakes in the German translation. Some of the texts are translated technically correct, but are a little bit off in the specific context. Maybe the guy who did the translations had some problems understanding the correct meaning given the base localization accompanied by some comments. To provide a little bit more context to the translator, you want to reorganize all your localization files so that for each string all three translations are grouped together. This way, he sees all three translations of the same text together and can spot differences easily.

An example for the translation of the phrase “got it”:

English: Got it
Spanish: Ok
German: ich habe es

The German translation is not optimal, “Verstanden” or “Ok” just like the Spanish translation would be better. This example may seem a little bit out of the air, but it demonstrates how localizations can be manipulated and how Logen can not only be used to create localization formats for a new platform, but also that it can help analyzing existing content.

Adding and implementing the new converter

We will begin by adding an empty file for the new converter to the converter module that holds all converter classes. We will name our new converter GroupingConverter. To do this navigate to the top level and execute:

touch Logen/converter/GroupingConverter.py

Given the new file as a starting point, let’s first add all needed imports.

1. We will need the class ConverterInterface, which is the base class for all converter.
2. Our four model classes representing a localization are needed as well.
3. Additionally we import some classes from typing to add type hints to our new converter.

Now we can start implementing our converter. The next step is to declare our new class and make it inherit from the ConverterInterface we imported as Base.

This base class defines some functions we need to implement. There are six functions in total, four of them providing metadata and two are the actual important functions. Let’s look at each function:

The first metadata function fileExtension defines the extension of files this converter can read and/or write. In this example we want to produce a simple .txt file.

The next function is identifier, which specifies an identifier with which the converter can be used from command line. This identifier will be passed to the main function via command line to select this converter when using its functionality. In this case I chose „grouping“.

The last two functions, which describe the converter, are importDescritpion and exportDescription. Each of them provides more detailed information about how the converter can be used.

Next we can work on the main functionality of a converter — importing and exporting localization files. Let’s start by implementing toIntermediate which gets passed a file path and returns an optional entity of IntermediateLocalizations. Optional is imported from typing and means, there may be cases where no IntermediateLocalization can be created from the given file path. Since GroupingConverter will create a file with reordered localizations, it does not support importing such a file to a localization. Thus we will just raise a NotImplementedError if a user tries to use this function of our converter.

The function fromIntermediate takes an IntermediateLocalization and returns a List of LocalizationFiles. Again List comes from typing. Let’s see how we can implement this function:

1. Before we can write the new content to a file, we need to restructure our data. An IntermediateLocalization contains a list of languages and each language has a list of entries. Since we want to group all translations by key, we add a new dictionary. Each element in this dictionary will consist of the key of a translation and a list of tuples. Each of this tuples in a list will have the languageIdentifier as well as the translation.
2. To create this dictionary, we loop over each language of the given IntermediateLocalization and again loop over each IntermediateEntry of the current language.
3. At this point we have access to the key and the value of each entry, as well as to the language identifier this entry belongs to. We can structure this data as described before and add it to our dictionary.
4. Now we can start to create the content of the final file, which will be created by adding new lines to a string variable. For this we loop over every item in the dictionary and deconstruct it to grab the key and the list of localization identifier and localized values. We can now write the key to the content and loop again over all items of the inner list to add them to the content as well.
5. Finally we can create the new localization file. Since this function is supposed to return a list of files, we append it to a new list and return this one.

Writing tests

Let’s test our new converter! First we need to create a new file:

touch Logen/tests/tests_GroupingConverter.py

Again we start by importing a whole bunch of stuff:
— module unittest
— various model classes and
— GroupingConverter of cause

But before we write a test, let’s add two methods to provide an IntermediateLocalization with multiple languages as input for GroupingConverter and another one to define the expected output, which we can check the result against.

The first helper creates an instance of IntermediateLocalization with three different IntermediateLanguage elements, each of which have two IntermediateEntry objects. The second one returns a LocalizationFile, which contains the same content as the IntermediateLocalization but in the expected format.

Last but not least let’s add the test case. We call fromIntermediate from our subject under test (GroupingConverter in this case) and compare the expected output with the result.

We can execute this test either with

python -m Logen.tests.tests_GroupingConverter

to run the single test case or with

python -m Logen.tests.run_all

to execute all tests for Logen. In both cases we should see that our test for the new converter succeeds.

Validating types

By now we have a working converter. Let’s make sure, the type hints we added were correct by using mypy. Assuming you are on top level of the project, type these two commands into your terminal:

mypy Logen/converter/GroupingConverter.py
mypy Logen/tests/tests_GroupingConverter.py

There should be no output on the console, meaning that our type hints were correct.

Using the new converter in main

There are just some missing steps until we can use GroupingConverter from the command line. Open the file main.py and add an import for our new converter.

Next we need to add this converter to the list of known converter. Replace the variable registeredConverter with following:

This variable lists all converter which will be used by the tool.

Trying out the new converter

GroupingConverter is finally ready to be used! Let’s add some example data and run our new converter on it. Here is a json file with some data:

This file is in a format, Logen’s JSONConverter can import. You can find this file in the testdata directory in tests. It defines three strings in two different localizations, thus we can check if our new converter did a good job. Execute following command:

python -m Logen.main convert Logen/tests/testdata/GroupingExample.json /just/some/path json grouping — dryRun

This starts the convert subcommand and uses JSONConverter to import the file. This will create an IntermediateLocalization that will be passed on to GroupingConverter, which will format it in the way we want. The destination path is irrelevant, because we added the option — dryRun, so the generated content will be printed to the command line.

That’s it! If you want to see the complete code, you can find it in the Logen repository. But since this converter is just an example and not quite production like, it exits in a separate branch example/GroupingConverter.

This was the last article in my series about Logen. There may be more article about updates and when new formats are supported. If you use Logen some day, I would really appreciate feedback. You can reach me via Twitter!

Thanks for reading!