Localization

[Blizzard]

I just remembered something that we haven't discussed properly. It's localization and translation of the editor. The easiest way to do this is to have a file with corresponding keys and values where you then substitute the keys in the code with the actual string values of the localization. I suggest the same format we use at Cateia. It's simple, easy to overview and easy to use.

Code: Example

KEY_GOES_HERE
{
VALUE_GOES_HERE
}

This line is not followed by a { character in a new line and hence it is regarded as comment. It's that simple.

Since implementing this all to work requires some little bit of work, we can leave it out for the RGSS version. But in the long run I'd like the editor to be easily translated. We can even include several files in this format, one for every language that people translate ARC to so people can switch languages in the editor whenever they want.

[Ryex]

I had already planed on implementing this at some point and in a very similar way too.  I'm all for it.

[Blizzard]

Eh, I just realized a small mistake there. Let me fix it in the first post.

[Ryex]

So, I was going to write support for this but I realized that there is one issue we should probably discuss. how will these file be accessed? as I see it there are two options, one when a translation is requested it opes the file and finds the value and two the entire file loaded into memory and stored as a hash. there are problems either way. the first will be way to slow and the second is taking up valuable memory. is there a way we could compromise the two?

[Blizzard]

Load it all on start up and put it into a hash. There's not point for opening/closing files during runtime each time you need a string (besides the fact that it's horribly slow).
A hash like that doesn't take much memory. If we have 10000 words (which is REALLY a lot) with an average word length of 10 letters (which is a lot as well), that's only 100 kB of memory. That's nothing. So we will preload all strings.

I suggest you check out how it's been done in aprilui. Check out Dataset.cpp. It basically loads all files from a specific folder location. If there's a 2 letter country code specified, it loads that subdirectory.



EDIT: Stupid Imageshack resized it. D: But you can still recognize the directory structure.

[Ryex]

hmm, ok then. I'll load the currently selected language into a hash then. switching languages will load the selected language and replace the hash. as English will be the default and built in it won't load a language file by default.

to use the locilisation system you'll

import Kernel.language._loc as _loc

and wrap strings with the method like so

_loc("this string will be localized")

the method will use the passed string as a key for the loaded loc file to return a localized version.

I'll write up a script to search through all the source files and find all the strings wrapped with _loc() to build a template loc file.

[Blizzard]

from System.Imports import *

#=============================================================================================
# LanguageParser
#---------------------------------------------------------------------------------------------
# Parser for localization files.
#=============================================================================================

class LanguageParser:
    
    def __init__(self):
        self.localizations = {}
    
    def load(self):
        folder = "./data/lang"
        for fileName in os.listdir(folder):
            if fileName.endswith(".lang"):
                language = fileName.replace(".lang", "")
                self.localizations[language] = self.parseFile(folder + "/" + fileName)

    def parseFile(self, filename):
        data = {}
        try:
            f = open(filename, "r")
            # the first line of a UTF file can be "screwed up"
            skip = 0
            line = f.readline()
            for i in line:
                if ord(i) < 128:
                    break
                skip += 1
            f.seek(skip)
            # read data
            string = f.read().replace("\r", "")
            f.close()
            # regular expressions are awesome
            matches = re.findall("(.+)\n\{\n((?:.|\n)+?)\n\}", string)
            for match in matches:
                string = CEGUI.String()
                string.assign(match[1])
                data[match[0]] = string
        
        except Exception, e:
            print "exception: " + str(e)
            trace_exception()
        
        return data
    
    def exists(self,key):
        try:
            self.localizations[key]
            return True
        except KeyError:
            return False
        
    def destroy(self):
        self.localizations = None
                
    def __setitem__(self,key,item):
        self.localizations[key] = item

    def __getitem__(self,key):
        try:
            return self.localizations[key]
        except KeyError:
            try:
                if self != texts:
                    return texts[key]
            except KeyError:
                pass
        return "< "" + key + "" not found >"

Just edit this code before you use it, please.

[Ryex]

umm... where did you get that?

[Blizzard]

It's a piece of code I made a long time ago. But you'll probably see a lot of stuff that won't work for us like the CEGUI references, the forcing that all files have a .lang extension etc. The multi-language support here is handled by using several files with different names where each file contains all strings for one language where what we need would be that we have different directories for each language that contain any number of files without any limitations on extensions. That's why I said you should edit it. It would have been a good deal of the the origin code of the one in Dataset.cpp if C++ supported regular expressions natively.