Anti-aliasing for bitmap

[azdesign]

I'm in the middle of working an animation script, which involve frequent rotate and stretching (scaling x or y). The result was horrible, jagged edges, even at very minor degree of rotation and scaling. Image become distorted. Is there any plugin, or script I can use to force anti-alias for bitmap ? Thanks in advance 

[G_G]

Not really. Any anti-aliasing script would most likely give you a ton of lag due to Ruby 1.8's slow interpreter. The only logical solution would be to add an anti-aliasing function to the underlying engine. But that's not possible since the RGSS Engine is closed source.

Your best bet would be to take your animations into photoshop and apply the rotation and anti-aliasing yourself.

[ForeverZer0]

Maybe try Win32API, get the window DC, and use GDI objects. That would be pretty difficult, though.

[azdesign]

Yes, that will be my last resort, editing the rotated images manually in photoshop. I'm not familiar with win32api but I'll give it a try. My recent experiment was to use 2 images. both are same images, the top layer has transparency so just like an overlay. It help improve the jagged edges but still does not meet my expectation. Why would enterbrain didn't release the remastered XP version ?

@ForeverZer0, where should I start learning Win32API ? do you have a recommendation for the site and the IDE/compiler ?

Thx for the feedback guys 

[Blizzard]

Actually the only thing you can do is get the final image on the screen from GDI over Win32API, you can't do much with individual Bitmap instances.

[G_G]

Oh! I completely forgot! You can actually create a library in C and pass an actual bitmap through it. In which you can modify each pixel. Some guy over at HBGames did it. I'll see if I can find the code he posted.

C Code

#include <cstdlib>
#include <ctime>
#include <ctype.h>
#include <math.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <windows.h>
 
BOOL APIENTRY DllMain( HMODULE hModule, DWORD  ul_reason_for_call, LPVOID lpReserved) {
    switch (ul_reason_for_call) {
        case DLL_PROCESS_ATTACH:
        case DLL_THREAD_ATTACH:
        case DLL_THREAD_DETACH:
        case DLL_PROCESS_DETACH:
        break;
    }
    return true;
}
 
typedef struct {
    DWORD flags;
    DWORD klass;
    void (*dmark) (void*);
    void (*dfree) (void*);
    double *data; //red is index 1, green is index 2, blue 3, alpha 0
} RGSSCOLOR;
 
typedef struct{
    DWORD unk1;
    DWORD unk2;
    BITMAPINFOHEADER *infoheader;
    RGSSCOLOR *firstRow;
    RGBQUAD *lastRow;
} RGSSBMINFO;
 
typedef struct{
    DWORD unk1;
    DWORD unk2;
    RGSSBMINFO *bminfo;
} BITMAPSTRUCT;
 
typedef struct{
    DWORD flags;
    DWORD klass;
    void (*dmark) (void*);
    void (*dfree) (void*);
    BITMAPSTRUCT *bm;
} RGSSBITMAP;
 
#define ASSERT(x) if(!x){DebugOut("Failed: %s: %d", #x, __LINE__);}
 
extern "C" _declspec (dllexport) BOOL AlienTwo(long object){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object<<1)) -> bm -> bminfo;
    DWORD rowsize;
    DWORD width, height;
    LPBYTE row, pixel1, pixel2, pixel3, pixel4;
    long x, y;
    double red, green, blue;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    rowsize = width * 4;
    row = (LPBYTE) (bitmap -> firstRow);
    for ( y = 0; y < (int) height; y++) {
        pixel1 = row;
        pixel2 = row;
        pixel3 = row;
        pixel4 = row;
        if((y >= 0) && ((y + 1) < (int) height)) {
            pixel2 -= rowsize;
            pixel3 -= rowsize;
        }
        for ( x = 0; x < (int) width; x++) {
            if(((y + 1) < (int) height) && ((x + 1) < (int) width)) {
                pixel2 += 4;
                pixel4 += 4;
                red = sqrt(pow(((double) pixel1[2] - pixel2[2]), 2) + pow(((double) pixel3[2] - pixel4[2]), 2));
                green = sqrt(pow(((double) pixel1[1] - pixel2[1]), 2) + pow(((double) pixel3[1] - pixel4[1]), 2));
                blue = sqrt(pow(((double) pixel1[0] - pixel2[0]), 2) + pow(((double) pixel3[0] - pixel4[0]), 2));
            }
            else if(((y + 1) >= (int) height) && ((x + 1) < (int) width)) {
                pixel2 += 4;
                pixel4 += 4;
                red = sqrt(pow((double) pixel1[2], 2) + pow((double) pixel4[2], 2));
                green = sqrt(pow((double) pixel1[1], 2) + pow((double) pixel4[1], 2));
                blue = sqrt(pow((double) pixel1[0], 2) + pow((double) pixel4[0], 2));
            }
            else if(((y + 1) < (int) height) && ((x + 1) >= (int) width)) {
                red = sqrt(pow((double) pixel1[2], 2) + pow((double) pixel3[2], 2));
                green = sqrt(pow((double) pixel1[1], 2) + pow((double) pixel3[1], 2));
                blue = sqrt(pow((double) pixel1[0], 2) + pow((double) pixel3[0], 2));
            }
            else {
                red = pixel1[2];
                green = pixel1[1];
                blue = pixel1[0];
            }
            if((red + green + blue) > (double) 60) {
                if((red >= green) && (red >= blue)) {
                    red += 20;
                }
                else if((green >= red) && (green >= blue)) {
                    green += 20;
                }
                else {
                    blue += 20;
                }
                if(red > 255) red = 255;
                if(green > 255) green = 255;
                if(blue > 255) blue = 255;
                pixel1[2] = (BYTE) red;
                pixel1[1] = (BYTE) green;
                pixel1[0] = (BYTE) blue;
            }
            else {
                pixel1[2] = 0;
                pixel1[1] = 0;
                pixel1[0] = 0;
            }
            pixel1 += 4;
            pixel3 += 4;
        }
        row -= rowsize;
    }
    return true;
}
 
extern "C" _declspec (dllexport) BOOL BlackAndWhite(long object){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object<<1)) -> bm -> bminfo;
    DWORD rowsize;
    DWORD width, height;
    LPBYTE row;
    long x, y;
    int shade;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    rowsize = width * 4;
    row = (LPBYTE) (bitmap -> firstRow);
    for ( y = 0; y < (int) height; y++) {
        LPBYTE thisrow = row;
        for ( x = 0; x < (int) width; x++) {
            shade = ((thisrow[2] + thisrow[1] + thisrow[0]) / 3);
            thisrow[2] = shade;
            thisrow[1] = shade;
            thisrow[0] = shade;
            thisrow += 4;
        }
        row -= rowsize;
    }
    return true;
}
 
extern "C" _declspec (dllexport) BOOL BlackAndWhite2(long object){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object<<1)) -> bm -> bminfo;
    DWORD rowsize;
    DWORD width, height;
    LPBYTE row;
    long x, y;
    int red, green, blue, shade;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    rowsize = width * 4;
    row = (LPBYTE)(bitmap -> firstRow);
    for ( y = 0; y < (int) height; y++) {
        LPBYTE thisrow = row;
        for ( x = 0; x < (int) width; x++) {
            red = (thisrow[1] - thisrow[0] + thisrow[1] + thisrow[2]);
            red = (red * thisrow[2] / 256);
            if(red > 255) red = 255;
            green = (thisrow[0] - thisrow[1] + thisrow[0] + thisrow[2]);
            green = (green * thisrow[1] / 256);
            if(green > 255) green = 255;
            blue = (thisrow[0] - thisrow[1] + thisrow[0] + thisrow[2]);
            blue = (blue * thisrow[0] / 256);
            if(blue > 255) blue = 255;
            shade = ((red + green + blue) / 3);
            thisrow[2] = shade;
            thisrow[1] = shade;
            thisrow[0] = shade;
            thisrow += 4;
        }
        row -= rowsize;
    }
    return true;
}
 
extern "C" _declspec (dllexport) BOOL Emboss(long object){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object<<1)) -> bm -> bminfo;
    DWORD rowsize;
    DWORD width, height;
    LPBYTE row;
    long x, y;
    int red, green, blue;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    rowsize = width * 4;
    row = (LPBYTE)(bitmap -> firstRow);
    for ( y = 0; y < (int) height; y++) {
        LPBYTE thisrow = row;
        LPBYTE nextrow = row;
        if((y > 0) && ((y + 1) < (int) height)) {
            nextrow -= (width * 4);
        }
        for ( x = 0; x < (int) width; x++) {
            if(((y + 1) < (int) height) && ((x + 1) < (int) width)) {
                nextrow += 4;
                red = (thisrow[2] - nextrow[2] + 128);
                green = (thisrow[1] - nextrow[1] + 128);
                blue = (thisrow[0] - nextrow[0] + 128);
            }
            else {
                red = (thisrow[2] + 128);
                green = (thisrow[1] + 128);
                blue = (thisrow[0] + 128);
            }
            red = abs(red);
            green = abs(green);
            blue = abs(blue);
            if(red > 255) red = 255;
            if(green > 255) green = 255;
            if(blue > 255) blue = 255;
            thisrow[2] = red;
            thisrow[1] = green;
            thisrow[0] = blue;
            thisrow += 4;
        }
        row -= rowsize;
    }
    return true;
}
 
extern "C" _declspec (dllexport) BOOL MinusTwo(long object){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object<<1)) -> bm -> bminfo;
    DWORD rowsize;
    DWORD width, height;
    LPBYTE row;
    long x, y;
    double color1, color2;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    rowsize = width * 4;
    row = (LPBYTE)(bitmap -> firstRow);
    for ( y = 0; y < (int) height; y++) {
        LPBYTE thisrow = row;
        LPBYTE nextrow = row;
        if((y > 0) && ((y + 1) < (int) height)) {
            nextrow -= (width * 4);
        }
        for ( x = 0; x < (int) width; x++) {
            if(((y + 1) < (int) height) && ((x + 1) < (int) width)) {
                nextrow += 4;
                color2 = (((nextrow[2] * 0.3) + (nextrow[1] * 0.6) + (nextrow[0] * 0.1)) / 10);
            }
            else {
                color2 = 0;
            }
            color1 = (((thisrow[2] * 0.3) + (thisrow[1] * 0.6) + (thisrow[0] * 0.1)) / 10);
            if((color1 - color2) > 0.2) {
                thisrow[2] = 0;
                thisrow[1] = 0;
                thisrow[0] = 0;
            }
            else {
                thisrow[2] = 255;
                thisrow[1] = 255;
                thisrow[0] = 255;
            }
            thisrow += 4;
        }
        row -= rowsize;
    }
    return true;
}
 
extern "C" _declspec (dllexport) BOOL Sepia(long object){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object<<1)) -> bm -> bminfo;
    DWORD rowsize;
    DWORD width, height;
    LPBYTE row;
    long x, y;
    double red, green, blue;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    rowsize = width * 4;
    row = (LPBYTE)(bitmap -> firstRow);
    for ( y = 0; y < (int) height; y++) {
        LPBYTE thisrow = row;
        for ( x = 0; x < (int) width; x++) {
            red = ((thisrow[2] * 0.393) + (thisrow[1] * 0.769) + (thisrow[0] * 0.189));
            green = ((thisrow[2] * 0.349) + (thisrow[1] * 0.686) + (thisrow[0] * 0.168));
            blue = ((thisrow[2] * 0.272) + (thisrow[1] * 0.534) + (thisrow[0] * 0.131));
            if(red > 255) red = 255;
            if(green > 255) green = 255;
            if(blue > 255) blue = 255;
            thisrow[2] = (BYTE) red;
            thisrow[1] = (BYTE) green;
            thisrow[0] = (BYTE) blue;
            thisrow += 4;
        }
        row -= rowsize;
    }
    return true;
}
 
extern "C" _declspec (dllexport) BOOL Sharpen(long object){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object<<1)) -> bm -> bminfo;
    DWORD rowsize;
    DWORD width, height;
    LPBYTE row;
    long x, y;
    double red, green, blue, ap;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    rowsize = width * 4;
    row = (LPBYTE)(bitmap -> firstRow);
    red = 0;
    green = 0;
    blue = 0;
    ap = 0.8;
    for ( y = 0; y < (int) height; y++) {
        LPBYTE thisrow = row;
        LPBYTE nextrow = row;
        if((y > 0) && ((y + 1) < (int) height)) {
            nextrow -= (width * 4);
        }
        for ( x = 0; x < (int) width; x++) {
            if(((y + 1) < (int) height) && ((x + 1) < (int) width)) {
                nextrow += 4;
                red += nextrow[2];
                green += nextrow[1];
                blue += nextrow[0];
            }
            red = (((thisrow[2] - (red / 3)) * ap) + thisrow[2]);
            green = (((thisrow[1] - (green / 3)) * ap) + thisrow[1]);
            blue = (((thisrow[0] - (blue / 3)) * ap) + thisrow[0]);
            if(red > 255) red = 255;
            if(green > 255) green = 255;
            if(blue > 255) blue = 255;
            if(red < 0) red = 0;
            if(green < 0) green = 0;
            if(blue < 0) blue = 0;
            thisrow[2] = (BYTE) red;
            thisrow[1] = (BYTE) green;
            thisrow[0] = (BYTE) blue;
            thisrow += 4;
        }
        row -= rowsize;
    }
    return true;
}
 
extern "C" _declspec (dllexport) BOOL Soften(long object){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object<<1)) -> bm -> bminfo;
    DWORD rowsize;
    DWORD width, height;
    LPBYTE row;
    long x, y, xoffset, yoffset;
    int red, green, blue;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    rowsize = width * 4;
    row = (LPBYTE) (bitmap -> firstRow);
    for ( y = 0; y < (int) height; y++) {
        LPBYTE thisrow = row;
        for ( x = 0; x < (int) width; x++) {
            red = 0;
            green = 0;
            blue = 0;
            for ( xoffset = -1; xoffset <= 1; xoffset++) {
                for ( yoffset = -1; yoffset <= 1; yoffset++) {
                    if ((y > 0) && ((y + yoffset) < (int) height) && ((y + yoffset) > 0) && ((x + xoffset) > 0) && ((x + xoffset) < (int) width)) {
                        LPBYTE newrow = thisrow;
                        newrow -= (yoffset * (width * 4));
                        newrow += (xoffset * 4);
                        red += newrow[2];
                        green += newrow[1];
                        blue += newrow[0];
                    }
                }
            }
            red /= 9;
            green /= 9;
            blue /= 9;
            thisrow[2] = red;
            thisrow[1] = green;
            thisrow[0] = blue;
            thisrow += 4;
        }
        row -= rowsize;
    }
    return true;
}
 
extern "C" _declspec (dllexport) BOOL Splatter(long object){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object<<1)) -> bm -> bminfo;
    DWORD rowsize;
    DWORD width, height;
    LPBYTE row;
    long x, y;
    int randomx, randomy;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    rowsize = width * 4;
    row = (LPBYTE) (bitmap -> firstRow);
    srand((unsigned)time(0));
    for ( y = 0; y < (int) height; y++) {
        LPBYTE thisrow = row;
        for ( x = 0; x < (int) width; x++) {
            randomx = ((rand() % 3) + 1);
            randomy = ((rand() % 3) + 1);
            if((y > 0) && ((y + randomy) < (int) height) && ((x + randomx) < (int) width)) {
                LPBYTE newrow = thisrow;
                newrow -= (randomy * (width * 4));
                newrow += (randomx * 4);
                thisrow[2] = newrow[2];
                thisrow[1] = newrow[1];
                thisrow[0] = newrow[0];
                thisrow[3] = newrow[3];
            }
            else {
                thisrow[2] = 0;
                thisrow[1] = 0;
                thisrow[0] = 0;
                thisrow[3] = 0;
            }
            thisrow += 4;
        }
        row -= rowsize;
    }
    return true;
}
 
extern "C" _declspec (dllexport) BOOL Swap(long object1, long object2, int index){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object1<<1)) -> bm -> bminfo;
    RGSSBMINFO *palette = ((RGSSBITMAP*) (object2<<1)) -> bm -> bminfo;
    DWORD rowsize, pRowSize;
    DWORD width, height, pWidth;
    LPBYTE row, originalColor, newColor;
    long x, y;
    int red, green, blue, oldRed, oldGreen, oldBlue;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    pWidth = palette -> infoheader -> biWidth;
    rowsize = width * 4;
    pRowSize = pWidth * 4;
    row = (LPBYTE) (bitmap -> firstRow);
    for ( y = 0; y < (int) height; y++) {
        LPBYTE thisrow = row;
        for ( x = 0; x < (int) width; x++) {
            red = thisrow[2];
            green = thisrow[1];
            blue = thisrow[0];
            originalColor = (LPBYTE) (palette -> firstRow);
            for ( int i = 0; i < (int) pWidth; i++) {
                oldRed = originalColor[2];
                oldGreen = originalColor[1];
                oldBlue = originalColor[0];
                if ( (red == oldRed) && (green == oldGreen) && (blue == oldBlue)) {
                    newColor = originalColor - (pRowSize * (index + 1));
                    thisrow[2] = newColor[2];
                    thisrow[1] = newColor[1];
                    thisrow[0] = newColor[0];
                }
                originalColor += 4;
                newColor += 4;
            }
            thisrow += 4;
        }
        row -= rowsize;
    }
    return true;
}
 
extern "C" _declspec (dllexport) BOOL Blank(long object){
    RGSSBMINFO *bitmap = ((RGSSBITMAP*) (object<<1)) -> bm -> bminfo;
    DWORD rowsize;
    DWORD width, height;
    LPBYTE row;
    long x, y;
    int red, green, blue;
    if(!bitmap) return false;
    width = bitmap -> infoheader -> biWidth;
    height = bitmap -> infoheader -> biHeight;
    rowsize = width * 4;
    row = (LPBYTE) (bitmap -> firstRow);
    for ( y = 0; y < (int) height; y++) {
        LPBYTE thisrow = row;
        for ( x = 0; x < (int) width; x++) {
            red = thisrow[2];
            green = thisrow[1];
            blue = thisrow[0];
            thisrow += 4;
        }
        row -= rowsize;
    }
    return true;
}

Usage

class Bitmap
  Sepia = Win32API.new('dll_file_here', 'Sepia', 'L', 'V')
  def sepia
    Sepia.call(self.__id__)
  end
end

I compiled the code myself. Here is the solution. (Built using Visual Studio 2010)
http://decisive-media.net/gameguy/RGSSBitmap.zip

Here's the example RMXP project.
http://decisive-media.net/gameguy/Project1.zip

All that's needed is to create a method for your anti-aliasing needs. Then implement it into RPG::Sprite and edit the animation method.

[azdesign]

Wow thanks. Whoever create that code must be a very skilled programmer. Thanks you for compiling it. Gonna try the methods. 

[Blizzard]

I was going to suggest a DLL, but I didn't think anybody would waste his/her time on something like this. ._.;

[ForeverZer0]

My personal opinion, if your game requires this stuff, RM may not be the engine for you.
RPG Maker is for simple 2D graphics, if you need something more advanced, such as 3D graphics or advanced effects, its counter-intuitive to not just use a more powerful engine that supports it natively. Its like using a small detail brush to paint a house when there are large, broad brushes made specifically for it sitting next to it. You are going to work harder and waste more time to accomplish it, when it would have been easier and created a better final product to just use the appropriate tool in the first place.

[Blizzard]

Let me just add here that ARC handles textures better and applies linear filtering.

[Metalgalamoth]

Any way to get the zips? the original website is down u.u

[KK20]

No, you'd have to build the DLL yourself. Why do you want this anyways? As far as I can tell, it's just a collection of bitmap manipulation methods that I think someone else made an actual script out of. Doesn't appear to resolve the original anti-aliasing question.

EDIT: Yeah it was F0 who did it.
https://forum.chaos-project.com/index.php/topic,14381.0.html