Extremely slow import
Exceter

I have two problems which both stem from the same underlying issue. That being, importing large number of files is extremely slow. I have a zip file with 50,000 images in it that I need to import. And, before you ask, yes, I need all of them.

Problem #1: At first, the import speed isn't bad but it gets slightly slower every file. At this point, it's been importing for almost a full day, it's at approximately 36,000 of 50,000 and each file takes 3-5 seconds to import. Is there a faster way to import large numbers of resources?

Problem #2: There is more than one of us who needs to work on this project and this means that we need to import/export project zips and send them back and forth because the file storage used for the game files isn't native and, therefore, cannot be added to a version control system. While undesireable, this would still be plausible if import could be completed in a reasonable time frame. Unfortunately, importing an exported source package is every bit as slow as importing the images in the first place. How can we share a single WADE project between multiple people?

What should we do?

All 11 Comments
Gio

Hi

First of all, yes the import process is slow, especially if you have a large number of files. Unfortunately it comes down to one of the following options:

1. Fast import (import files in parallel) that uses lots of memory and is likely to crash Chrome

2. Slow import (import one by one) that uses much less memory and is unlikely to crash Chrome

We used to have number 1, then switched to number 2. I still think it's the better option. I will definitely look into ways of speeding it up though.

Have you checked that you have allocated enough space for your files? If you haven't changed the settings, the default option is to reserve 200MB for the file system. This probably won't be enough for your 50k images, and if it's failing to write files to your file system, then you'll have to do the whole thing again, which is not going to be fun! Also if it's trying and failing to write files, this may result in the import process being especially slow.

As a side note, make sure that the amount of disk space that you reserve for WADE is not more than 10% of your overall free disk space, otherwise Chrome may decide to randomly delete files to free up space.

To answer your other question, yes it is possible to use the same files collaboratively with other users via Dropbox or similar file-sharing solutions. Please see this thread.

Exceter

What about a slightly different approach... Is it possible to reference images via URL? If that were possible, I could push them to Amazon Cloudfront and I wouldn't have to import them at all.

Gio

Yes sure, instead of using a relative file name such as "myImage.png" you can simply use "http://example.com/myImage.png".

Be aware though, that this may limit some of the things you can do with the images because of cross-origin restrictions. For example, if you want to use pixel-perfect mouse events (which requires reading the alpha of each pixel in the image), CORS restrictions may prevent you from doing that.

Starting from WADE 4.0, all images loaded via wade.loadImage will get an "anonymous" crossOrigin attribute, which should help mitigate that problem.

Exceter

Is this a problem which would only affect the online editor or would it also affect exported game clients?

Gio

Loading images from a different domain is always going to have problems. Whoever designed the HTML5 specs thought this was a good idea, for security.

But once you publish your game, presumably your images will be on the same domain as your other app files, so there should be no problems there.

Gio

Now I don't know if this applies to your case, but I'll say this generally for people that may be in the same situation as you.

If you're having to deal with a very large number of (shared) assets, I think it'd be easier to do this:

1. Develop your core gameplay in the editor using a small-ish number of assets

2. Export everything and work offline using a separate IDE such as WebStorm to test your game with all the 50k images that you need.

Having said that, I've been working on a WADE game that uses about 1200 images. Yes it takes a while (10 minutes or so) to import / export, but other than that the editor can cope with it. If I had 50,000 images rather than 1,200, I'd do what I just said above.

Exceter

What are the disadvantages of working offline in a different IDE? Is testing harder?

 

Gio

Well the advantage of working in the editor is that you can drag and drop things, draw your own paths and visualize them, resize and adjust things until they look right, see what your shaders do as you type, etc.

But normally, you'd do that with a limited number of assets.

Say for example that I'm making a match-3 game that uses 10,000 possible images for my items. I'd work in the editor to make it work with 5 or 10 different items, tweaking the size of each cell, the movement speed, particle animations and all that.

Then when I'm happy that it looks nice, I'd export it and change the code so it loads 10,000 different images instead of 5.

Gio

To elaborate, you can also keep the "small" version of the game in your editor and the bigger one offline, provided you remember to keep changes in sync. For example if you change the UI in the editor, you'll want to download your scene.wsc file and copy it to your full game folder.

Teecee1337

I'm working with Exceter on this and I've been testing around in the WADE but i'm having some problems. I'm new in scripting, so i'm trying my best.

1) I see your test files with animations is 1 PNG (the sheep & the swordsman). Our PNG's are all invidual files of actions (Eg. Files 001 - 008 are actions of char_walk_se. We need to change the invidual PNG's to 1 i'm guessing ? Is there a program to do this ?

2) The quality of the IsoChar is very poor when I put it in my 'game world'. The PNG's are 120 x 200 with 32 bitdeep,
Some PNG's are even higher 240 x 350, but still i'm getting a really small image and resizing them makes them blurr
Is there a possibility to fix this or am I doing something wrong ?



 

Gio

Yes to use animations in WADE, you need all the frames in the same png. In fact there may be an argument to say that, for performance reasons, you should also put as many animations as possible into the same PNG, and use the minUV and maxUV parameters to define which area of the png the animation is using. However this is an optimisation and you can worry about that later. To start with, you can have one PNG for each animation.

There are several ways of doing this. The important thing is that all the frames of the same animation must be the same size. If you have currently got an individual png for each frame, then you want to first make sure that you resize each frame so all frames of the same animation are the same size.

I wouldn't recommend using Photoshop or similar tools to do this. It's much easier with a command-line tool such as imagemagick

Once downloaded and installed, you can do something like

magick fileName.png -background transparent -gravity center -extent 200x200 outputFile.png

This will create a file called outputFile.png with your image resized to 200x200

Once they're all the right size, you want to put them next to each other. Again, if you google a bit you'll find a way to do it with imageMagick. Alternatively you can use something like TexturePacker

Regarding your second point, even 240x350 is not very high resolution. If you use a power-of-two for the size of your image though, when you resize it your graphics card will be able to use bilinear filtering, so it'd look a bit better. Still not as good as a high-resolution image, but a bit better.

Post a reply
Add Attachment
Submit Reply
Login to Reply