Encrypted Archives

[G_G]

First of all, the "or" in the description of this board is meant to be "of" if I'm not mistaken. Second, how are we going to be creating our encrypted archives. I sure as hell don't want it to be as easily decrypted as RM games are right now. Though I'm sure someone (vgvgf) will figure it out eventually.

[Blizzard]

I really make that mistake often. ._.

Everything can be figured out, nothing is unhackable.

As for the encryption, somebody should first research what file encryption methods and concrete algorithms there are. Then we will figure out a way to implement an asymmetric algorithm and this should do the trick. Don't even bother with anything less powerful than asymmetric algorithms. Any volunteers for the research?

[G_G]

I'll look into it but I can't guarantee I'll be of much help. First of all, file encryption, randomizing, etc... is probably one of my weaker points as I never really studied it. And I'm not that great of a researcher either.

[ForeverZer0]

How about the RSA Algorithm? At first look, it seems it could be a possible candidate for our encryption method.

I know you want to stick with an asymmetric algorithm, but as far I know the Triple DES Encryption has proven to be quite secure. In fact, it is the standard for ATM's and other forms of money transfer in the world. As far as practical implementation goes, that I am still uncertain of.

[Ryex]

I thought DES was asymmetric.

in any case I have no problem with RSA it's secure and as long as we build the Zer0Division.dll with the private key obfuscated then it is unlikely that people will be able to crack it.
the bigest mistake Enterbrain made was using symmetric encryption and having the key blatantly defined in the dll.  if we say had the private key generated through several function calls people wouldn't be able to find it any where near as easily.  we just need to keep in mind that asymmetric encryption is much slower than symmetric.

[Blizzard]

As far as I know AES is better than DES and I know how we always make fun of AES being the weakest encryption. O_o
Take a look at this: http://en.wikipedia.org/wiki/Block_cipher_security_summary

http://en.wikipedia.org/wiki/List_of_algorithms#Cryptography
http://en.wikipedia.org/wiki/Topics_in_cryptography#Modern

RSA is public domain now so it might be a good choice. But let's first check out what else is out there before we make a final decision.

[G_G]

Alright, I think I'm stuck with having individual encrypted files. Why? Easily make patches and updates. Easily see if a file exists or not. I mean it seems right doesn't it?

[Blizzard]

Yeah, that was my first thought as well. If we make archives, then we also have to use program the access to those archives and we have to mess around with path changes, etc. It's way too much work for something that is actually a worse solution. I also think we should stick to individual files.

[Ryex]

I agree this way we can have the encrypter run on directories encrypting every file it finds. and we could do what RGSS did and have the load_data method automatically unencrypted encrypted files

[Blizzard]

Yup, that's the idea.

[Ryex]

http://en.wikipedia.org/wiki/Elliptic_curve_cryptography

http://en.wikipedia.org/wiki/Crypto%2B%2B

an alternative to RSA?

[ForeverZer0]

It would appear that the library from the second link has a lot to choose from, both elliptic curve and RSA. I think this is a great to add to the project.

As for the actual method to use, I like them both. I admit that I am not super-versed when it comes to encryption methods, but both seem like they could fit our needs. It's a flip of a coin for me.

[Blizzard]

What about the license? If the license is fine, I think we can go about using another external library rather than coding it ourselves.

[Ryex]

http://www.cryptopp.com/License.txt

the whole compilation is open source and the individual files are public domain. I see nothing is the license that prevents us from using it.

[Blizzard]

This is great. It will allow us to offer an encryption option for the end user before compiling the project into a game distribution. xD

[Ryex]

if we offer that service..

[G_G]

Just trying to clear things up before I look stupid again.

We're basically going to have 2 formats right? One for the Engine and Editor to load. Basically the .rxdata. Then we'll have our encrypted format that only the game engine can load.

I just wanted to throw a suggestion out, have a hidden method that saves data in our encrypted form. Obviously we'll have "load_data" to load the encrypted data, but a save_data method would be nice as well.

Also, do we plan on encrypting Audio files this time?

[Blizzard]

Yes, yes and yes.
.rxdata is actually just serialized Marshal, we will have our own format for that.
We will have a save_data method which will be also visible to scripters in case they want to take advantage of the format. It will be basically the msgpack format.
And yeah, we will implement audio encryption. In fact, upon "compilation", you will be able to choose what stuff you want to encrypt.

[Blizzard]

Alright, guys, we're getting close to an actual release here. I wanted to bump this topic again so we can resume this discussion and find an appropriate solution.

We will support ARC::Data.load and ARC::Data.dump for our own format and load_data will do the same thing that RMXP does. This is for script compatibility.

As for the encrypted format, we will use the .arc format for the data which is used by the editor and we will call the encrypted version .arc as well. Files names will not be changed, only the extension will be changed.

If we can pull it off anyhow, we should try to generate a huge encryption key based on something the userdeveloper gives us. e.g. he could give us his name and we use a hashing function to generate some badass key that will be used to encrypt his game individually. The question is only how we can pull this off since we have to add that key somehow into the game so the engine knows what to use to decrypt the data. Or should we use one key? Because if we allow a different key, updates for any games would require the user to enter the same string for the key generation which might not be such a good idea. Except, of course, if the encryption key generation string becomes a project setting that is saved so the user can compile his game with the same encryption key as much as he wants.

[G_G]

The encryption should definitely be saved in the project settings if we go this route. As far as how we're going to save the key into the engine is beyond me. If we just use one key for everyones game, chances are that someones going to find out that key eventually.

[Blizzard]

If we use a fixed size key and can compile force it into the DLL to reserve us zero-bytes across the DLL that we will literally fill up during compilation... we will still make it easy for hackers because they can just compare 2 DLLs, find the differences and extract the keys for both. ._. Any ideas on how we could handle this? The problem here is that we need to hide the public key of the encrypted data. :/

[Ryex]

is there any way we can get the dll to 'Generate' the key instead of holding it internally and passing it along? I know it not as simple as putting a few function in because compilation will tempt to condense those function down to their result.

[Blizzard]

We should inform ourselves properly about obfuscation. I just took a quick look around the Internet and one of the methods to do this is to generate the key based on function calls, "dead blocks" and other stuff. Check this quick slide presentation out: http://logic.pdmi.ras.ru/~yura/en/stuttgart-talk.pdf You should also check out the links given at the end of the presentation. We will most likely have to add just one key into the DLL and hope nobody cracks it. :/ There has to be a good way to hide the key.

[Ryex]

Ok, I just had an idea. In my head it seems to make it near impossible for someone to obtain the data from the files with out reverse engineering ARC to get the source code.

first things first:
only the game exe should be able to encrypt and decrypt data.
when ARCed generates an encrypted archive it takes a user string generates a numerical sequence and uses that sequence in a process call to the ARC exe.

so ie

Game.exe -e list|of|files|to|encrypt 1754932017649

there is no reverse of this to get the un-encrypted files only the load_data function tied into the ruby scripts can access the decryption function

2)
the encryption key is never stored anywhere in it's full form.

3)
to get the key we call a sequence of functions that when called in a specified sequence generate a key. the sequence passed to the encrypter tells it the sequence to call these functions in to get the key.

4)
the function used in the decryption process are never publicly available in the dll so linking to it will NOT give you access to call these functions. the functions will only be called internally when a file is recognized to be encrypted

5)
now we store this sequence somewhere. while this sequence IS easily obtained like the DEADCAFE key in RGSS was with it alone it is impossible to obtain the real keys for the decryption with out reveres engineering the set of functions that are called to make the key. this IS possible but only for a significantly motivated and extremely knowledgeable individual. we can the task harder for them and thus less worth it by using a code obfuscator to make following these function calls in the byte code (thus finding the byte code that needs to be reverse engineered) more difficult.

I'm not sure if that made sense to you guys but in short this method obfuscates the key quite significantly form the public AND allows uses to have unique encryption keys


EDIT:

this system is only secure if all encrypted files are combined into an archive of some sort or the scripts file made so it can't be replaced. other wise a fresh script file containing instructions to load and dump the encrypted files in non encrypted form could be encrypted with the sequence which would be easy to find and replace the existing scripts file with their freshly encrypted one so as to "inject unsafe code" like a virus. once the code is in there the encryption key doesn't matter.

this is a major flaw. perhaps you guys can take parts of this idea or build on it to remove this flaw.

[ForeverZer0]

I use {smartassembly} for .NET projects, and it has a setting for a dynamic proxy. The proxy is created dynamically at run-time, and all external and public calls are made through it. It supposedly prevents any type of post-build modification, because it screws up the proxy, rendering it useless.  I don't really know how this achieved, nor if we could implement such a thing in ARC, but it may be another avenue to look at.

[Blizzard]

I was thinking to obfuscate a hardcoded key the same way Ryex already described, to generate it through a series of function calls. In fact I was thinking of a seriously long key (like 128 characters) where we spread the functions across classes to make things every more difficult. We can mess even with the decryption by using a function pointer and passing it across some classes before we actually call it. This is nasty to reverse engineer and we should be able to deflect the simplest attacks to keep out 99% of the hackers. I think we shouldn't implement the encryption function in the DLL at all. ARC::Data.dump is fine, but without the encryption.

The scripts will be in some sort of archive (other files won't), but we have to figure out a way how to disallow simple replacement. I see no other way than generating some sort of hash or fingerprint. But then again we have to problem where to put that one and we are back at square one where we would have to modify the DLL. :/ The only way I can think of is that we generate a file with a special key that is generated together by using all files included in the project and the DLL won't load a scripts file if it was replaced because it couldn't have been generated with the same files. Even if the hacker replaces the key file, he won't be able to load the scripts as the key file will not work with the other files (because it regenerates that hash key using the data files so they key won't match again).

[Ryex]

oh, so it would require a key file that contained a hash created from all the files that were encrypted together so that if a file was replaced the resulting fingerprint of the file together wouldn't match. This idea sounds like it could work, but it suddenly makes it hard for the creator to update individual files... unless of course the dll contains a way to modify the key file correctly so that file contents could be replaces if it was done from within the game engine.

[Blizzard]

Not necessarily. The creator just has to make sure he ships all the changed files and the file with the new hash key.

[G_G]

Well maybe we should just have 1 single key then, like you said created globally across several functions and classes. I bet we could combine 20 sex terms for the key and no one would ever guess it. :V
In all seriousness though, if allowing people to have their own key is too difficult, one key would be fine, we just have to make it more secure and longer then DEADCAFE.

[Blizzard]

Let me explain: We need two different keys. One is the decryption key to protect the data. That one can will be generated through a series of functions, that's the DEADCAFE equivalent of RMXP's key. The other is a hash key to make sure nobody replaces the scripts with their own and accesses the decryption method from the outside. Basically a hacker would have to both find the DEADCAFE equivalent key, replicate the decryption algorithm or replicate the hashing algorithm to decrypt the resources.

[Ryex]

It sounds like a pretty solid idea. but can we do it? do we have the parts to rebuild him?

[Blizzard]

Who cares about how RMXP does it. We're probably going with RSA. We'll just obfuscate it a bit here and there by spreading it across functions and methods within other classes and we'll mess around with function pointers. It will not be impregnable, but it will keep out 99% of the hackers (not noobs).

[Ryex]

nothing is impregnable, the point is just to make it too hard for anyone but a skilled and VERY motivated person to feels it's worth it and put forth the effort.

[Blizzard]

Exactly. If somebody cracks it, at least we know somebody who wants to learn and is dedicated did it and not some smart-ass thief.

[Ryex]

I had a small idea. should we perhaps package two game exes with ARC? one debug exe that can only run from un-encrypted files and accepts the -debug command line argument. and one that doesn't accept arguments and can only run form encrypted archives with a proper hash key that was built with the archive? this might make it easier to make, and perhaps harder to crack.
I'm kinda sleep deprived at the moment so I might just be being stupid.

[Blizzard]

We can just compile 2 exes without the need for an additional argument. One is the debug, the other is the release one.

[G_G]

I think ryex wants the -debug argument in case people want to run their game without being in debug mode.

[Blizzard]

I had a small idea. should we perhaps package two game exes with ARC?

[Ryex]

the difference would be that one of the exes counld only run unencrypted files and could accept a -debug argument.
the other exe would NOT accept the  -debug argument and would only run from encrypted archives

[Blizzard]

I just remembered something. Do you know what package signing is? I mean this would be pretty much what we're trying to do here. Maybe it would be wiser if checked out how others did it and see if we can learn something or gather interesting information.

[Ryex]

I know it is used to authenticate assemblies. I've no idea how it works or what it entails.

[Blizzard]

It's very simple. We just calculate a hash (e.g. MD5, SHA-1, SHA-2, etc.) and store the value somewhere. That way the package can't be edited without breaking the game because we calculate the hash for validation whether it was changed.

[Ryex]

well, you clearly know more about the subject than any one else on the team . I say that you com up with a way to implement the system. and if you need any thing in the editor let me know and I'll add it.

as far as I understand it we are going to encrypt the files and then sign them with a hash computed from the files. the hash will either be stored in a key file or attached to the  encrypted scripts file so that the engine can verify that nothing has been modified before running the game.

it sounds simple enough in principle and I'm not seeing any easily exploitable flaws. I say we go for it.

[Blizzard]

Yeah, it's fairly simple. I was thinking of encrypting the file first and then creating a hash which is the simply appended at the end of the file (all done in Python during project compilation). When we read the file in the engine, we read the whole file and separate the hash at the end. Then we create a hash value in C++ and check if we got the same one as the one that was written at the end of the file. Finally, if the hashes match, we decrypt the data and we have the original file.

http://en.wikipedia.org/wiki/SHA-1
http://en.wikipedia.org/wiki/SHA-2

While SHA-1 is relatively safe, it was cracked with a theoretical attack. WIth SHA-2 we also have the option of different lengths (SHA224, SHA256, SHA384 or SHA512). For the C++ version, we can just take the code from Crypto++ which is public domain.

EDIT: I didn't take time into account. Maybe SHA-1 would be better suited for our needs after all.

[Ryex]

SHA-1 is natively supported in python http://docs.python.org/library/hashlib.html#module-hashlib

so that may be best.

[Blizzard]

Then lets make our own lives a bit easier and just use SHA-1 for hashing. xD

[Ryex]

Yeah, it's fairly simple. I was thinking of encrypting the file first and then creating a hash which is the simply appended at the end of the file (all done in Python during project compilation).

I'd just like to point out that Python byte code is easily reverse engineered. much more so than a C++ assembly. so we need to do an elaborate method of obscuring the encryption key there too.

[Blizzard]

Yes, we still have to think of a good encryption.

EDIT: Lol, a collegue at work sent me this link completely unrelated: http://www.inner-smile.com/nocrack.phtml

[Ryex]

We can;t put the encryption in the editor period we have to keep it in the game exe where it can be protected.

all members in python are public. a clever hacker could create a editor plugin that would bare the name space. if they searched long enough they would eventually find the component used for encryption.

and this exists http://sourceforge.net/projects/unpyc/
it translates python byte code to readable python code. so encryption in python is out period.

we should just make it so that running the game.exe with a -e arguments causes it to take the game data and encrypt it. if we want to be able to control which files get encrypted we could write a configuration file with the editor which the exe loads when it is stared as an encryptor.

[Blizzard]

I am against putting it in the game exe as well. Then hackers would just need the Game.exe to get everything. One of the first things you do when you have to use encryption is taht you don' tput both the encryption and decryption code in the same place. And giving access through -e would basically render the whole thing pointless. Anybody could change the data files whenever they want. If we want to make it safer in the editor, I suggest a small .dll written in C that Python can load and use to encrypt.

[Ryex]

that could work I guess. the c dll that is. I really don't have any clue how to  write a C dll for python though

[Blizzard]

You just write the C DLL and compile it (careful, NOT a C++ DLL!). You should be able to easily call the DLL functions through Python just like in Ruby.

Oh, also, here's the Python code for the getting of environment variables that I said I'd give you. You probably call C DLL functions the same way.

    def getEnvironmentVariable(self, name):
        n = ctypes.windll.kernel32.GetEnvironmentVariableW(name, None, 0)
        if n == 0:
            return None
        buf = ctypes.create_unicode_buffer(u'\0' * 10000)
        ctypes.windll.kernel32.GetEnvironmentVariableW(name, buf, n)
        return buf.value

Don't forget to import ctypes.

[G_G]

Perhaps I could actually help on this part a bit. When I was doing that Xbox to RGSS thing I was creating my own C DLL. First you need to create a Win32 project in Visual Studio. When the Wizard comes up click Next, then choose DLL and click Finish. When creating methods for this you have to export the methods to C. So I assume you shouldn't need a whole lot of methods.

extern "C" _declspec (dllexport) BOOL Encrypt()
{
    // insert encryption code here
}

Then you'd be able to access that Encrypt function.

[Blizzard]

The function should take 2 arguments: (unsigned char* stream, int streamLength)

[Blizzard]

Reviving this topic one last time.

Ryex told me that he can encrypt and obfuscate the final .exe and Python code for the editor which basically means we can implement the encryptor in Python without problem. I will take care of that as I will also implement the decryptor in the engine. This means that things are simplified a lot. These are the procedures.

Encryption (Python):
1. generate a "static" encryption key using several method calls (for obfuscation, just to be sure) where the actual private key will be preselected (done only once)
2. read file (from here on, this is done for every file)
3. encrypt read data
4. create SHA-1 hash from encrypted data and append it to the end of the encrypted data stream
5. add 128 to the first byte (HURR HURR HURR)
6. write it all in a final file with .ade (data), .aae (audio) or .age (graphic) extension.

Decryption (C++):
1. generate a "static" decryption key using several method calls (for obfuscation, just to be sure) where the actual public key will be preselected matching the private key in the encryptor (again done only once)
2. read encrypted .ade, .aae or .age file (again done for every fiel)
3. add 128 to the first byte (same step for encryption and decryption HURR HURR HURR LITTLE HACK)
4. read file data, separating data and SHA-1 hash
5. create SHA-1 hash from encrypted data and compare it to the previously stored SHA-1 hash (check if data was messed with)
6. decrypt the data using the public key
7. write a temporary file, load the data and delete the temporary file by first overwriting it (simply opening it again in 'wb' mode) and then deleting it properly

As for the scripts encryption...
Obviously the scripts will be encrypted with RSA and hashed with SHA-1 as well. But additionally to that we will create a special key file that is shipped with the game. In this key file we will store the hash values for all scripts + each file. Basically we will have a file that has the hashes for every file combined with the main scripts file. This will result in as many keys as there are files. When we load the game, we randomly select a couple of these keys (10 or 20) and calculate those few hashes again. This will prevent a big loading overhead. If everything is fine, nobody has tempered with the scripts file. If it is not fine, somebody was messing with it (or with the given files) and the application will simply close (no error message or anything!).

This is the final solution!

And the previous note is there so I don't have to go through the whole topic the next time to find it.

[Ryex]

the python obfuscation is mostly a by product of the python code being translated to C++ code with calls into the python run time to do everything. if someone can understand it after it has been compiled then they are a god of reverse engineering.

[Blizzard]

Also, the file with the keys will be the misleading "libarc.dll". HURR HURR HURR. If somebody asks what it does or why it differs in size among games, we just tell them that this is a file used for some specific game optimizations. :3

[Ryex]

you tricky tricky bastard you.

[Blizzard]

We must be awesome, because we reinvented something Steam did before us without knowing that Steam did it. :3

[13:10] <~Blizzard> oh, BTW
[13:10] <~Blizzard> we thought of a really cool way to protect ARC's game files
[13:10] <~Blizzard> you know, so nobody except the game can decrypt the stuff
[13:10] <~Blizzard> and so nobody can replace the scripts file with their own :3
[13:12] <Zeriab> =o?
[13:12] * Zeriab is interested
[13:12] <~Blizzard> I can't really say much, obviously
[13:12] <~Blizzard> but let me phrase it like this
[13:13] <Zeriab> Are you going the Steam way of allowing individual encryptions which you can give to each person?
[13:13] <~Blizzard> the game will know if a file has been modified
[13:13] <~Blizzard> no, that would be too much
[13:13] <Zeriab> Oh, nice
[13:13] <~Blizzard> we have a universal system
[13:13] <~Blizzard> that will "encrypt" a few things depending on the game data
[13:14] <~Blizzard> you could say that no game will have the same "encryption pattern"
[13:14] <~Blizzard> and this encryption pattern will protect the game data
[13:14] <Zeriab> awesome, so it is a bit like what steam does :3
[13:14] <~Blizzard> this will not only disallow editing externally
[13:14] <~Blizzard> it will also prevent people from inserting their own scripts file to extract the data
[13:14] <~Blizzard> really? xD
[13:15] <~Blizzard> I have no idea how they do it xD

[Ryex]

just a question, is it possible that we could use my CycleByte method as the base for the encryption? it uses SHA-1 and it's cryptography secure as far as I can tell it would be a fairly simple to implement it.

the only thing is that it uses the same key for encryption and decryption and I've no clue how to make it otherwise.

[Ryex]

So, is this still a valid solution what with the opensource deal?

[Blizzard]

Well, yes and no. We can still offer a way to encrypt things, but it can be reverse engineered easily. So there is actually no point anymore.

[Ryex]

perhaps we should provide it as a closed source plugin. it would be simple enough to provide a dll that could be loaded on startup if its' detected that could preempt the load process for the game yes? hook into the load methods for data and decrypt files before passing them on to be loaded?

[Blizzard]

Yeah, that would be a better solution. But let's drop it for now.