Week of 7/30/2014-8/5/2014

It has been the final week as a summer research assistant!

• Summary of the project 

I wrote a summary document of the project and it is available here.
Also, an interactive demo is available here.

• Knitting application 

 A knitting application is implemented as follows. This application uses a lot of small 3D objects, so it is a  good performance benchmark. It draws 11 * 21 * 2 = 462 cones in about 3 seconds on my laptop (ThinkPad T430s).

This application consumes about 30% of CPU even after the rendering and makes the user interface somewhat slow. To speed up the rendering speed, we may need to use WebGLRenderer instead of CanvasRenderer, which we are currently using.

Week of 7/23/2014 - 7/29/2014

• Creating common 3D shapes using 3D pen

 To let users stamp common 3D shapes, the following codelets (blocks) are added in the "Pen" palette.

Here are some example shapes created from the above codelets:

A "knot" codelet is intended to be used for a 3D version of the knitting application (http://community.csdt.rpi.edu/projects/38/). For the detailed parameter usage of the knit codelet, please refer to this interactive example (http://www.smartjava.org/ltjs/chapter-05/08-basic-3d-geometries-torus-knot.html).

Currently, stampled shapes take over the following properties from a 3D sprite (in above example, a bird): color, position, rotation, scale, and texture. Thus, if you want to change those properties for shapes created from 3D pen, first you can apply 3D operations to the 3D sprite and then stamp any shapes you want.

Using the arc codelet, we can develop a VirtualWigwam application as follows:

Hereby we achieved the goal we set in the beginning of the summer!

• 3D shape display control for speed up the rendering

In the current design, the rendering process gets increasingly slow, when you add more 3D shapes. For example, when we want to display N arcs created by 3D pen, rendering steps go like the following:

- 1st step: rendering 1st arc
- 2nd step: rendering 1st and 2nd arcs
- 3rd step: rendering 1st, 2nd, and 3rd arcs
   ....
- Nth step: rendering 1st, 2nd, ..., and Nth arcs

Thus, it costs you N*(N+1)/2 times of rendering the arc. This is unavoidable because if we change anything in the 3D space and make it visible, we need to request Three.js to update the entire canvas for 3D. Three.js might be doing something intelligent to avoid rendering the same objects over and over again, but at least the above sequence is what we request from the application's perspective.

Having said that, if we do not need to display intermediate results from the 1st - N-1th renderings, we can just skip these steps and display the Nth result only. Based on that idea, I have implemented a "hide pen" and a "show pen" codelets, which stops and resumes rendering 3D shapes by 3D pen.
Using these codelets just as the following, the system just renders arcs N times, not N*(N+1)/2 times.
 
  hide pen
  repeat N
    arc width: 100 height: 100
  show pen
 

• Switching codelets depending on if the sprite is 2D or 3D

 There are codelets that are applicable only for 2D, or 3D. So depending on the status of the sprite (i.e., 2D or 3D), showing codelets that work for the selected sprite makes sense. For example, when a 2D sprite is selected, the "pen" palette shows the following codelets:

 

On the other hand, when a 3D sprite is selected the same palette looks as follows:

 

• Concluding remarks

My summer job ends on August 1st. For the rest of the week, I will work on a document that describes the implementation of the 3D system for C-Snap.

I have been really enjoying this project and am reasonably happy for the achievements I made including the VirtualWigwam.

Hope someone takes over my work and further enhances this project!

Week of 7/16/2014 - 7/22/2014

• Texture mapping

Texture mapping is to apply an image to all the surface of a 3D object. Any regular images such as JPEG, PNG, GIF, ..., etc. can be textures. After loading a 3D costume of your choice, a texture can be selected from the newly added "Textures..." menu as follows:

  

Here are some examples of 3D objects with textures:

• Switch from 2D to 3D

After implemented the texture mapping function, I realized it was easy to turn 2D objects into 3D  by applying a texture to a 3D plane. So, I added a new switch to bring 2D objects into the 3D space  as follows:

Here are some examples of 2D objects in the 3D space:

 A dog is created from a 2D costume whereas a smiley is created from a user-painted image.

• To Do

Here is the updated To Do list:

    1. Drag & drop
    2. Nesting
    3. Thumbnail display
    4.  ✔ Overlapping with 2D object
    5.  ✔ Transformation (move x y z)
    6.  ✔ Rotation (rotate n degrees in x, m in y, l in z)
    7.  ✔ Scaling
    8.  ✔ Textures (apply this image to all the surface)
    9. 3D pen (render common 3D shapes such as cube, arc, sphere, cone, ..., and so on)
By implementing the switch from 2D to 3D, 4 is actually done. Next week, I'll be working on 9 and that would conclude my summer job.

Weeks of 7/9/2014 - 7/15/2014

I have been trying to bring "real 3D" to C-Snap since last week, and now it works great.

• New implementation 

In the new implementation, one layer is dedicated exclusively for 3D objects as follows:

Three.js takes full control of the 3D layer so that basically we can do whatever three.js can do.  We lose some functions that used to be working (will be explained below), but take advantage of great 3D capabilities directly coming from three.js. An example of rendered 3D objects is shown below:

As you can see, there are grid lines, which can be turned on and off by the following toggle switch associated with the stage:

Moreover, we can change the position of the camera using the following blocks newly added for the stage. Using this blocks, we can change the perspective of the 3D space easily.

• Functions not working anymore

So the real 3D starts working fine, but there are some drawbacks. So far, I have noticed the following functions are not working anymore with the new implementation and needs to be implemented.
      1. Drag & drop
      2. Nesting
      3. Thumbnail display
      4. Overlapping with 2D objects

• 3D functions to be implemented

Aside from the above four functions, there are some more functions essentially for 3D:
      5.  ✔ Transformation (move x y z)
      6.  ✔ Rotation (rotate n degrees in x, m in y, l in z)
      7.  ✔ Scaling
      8. Textures (apply this image to all the surface)
      9. 3D pen (render common 3D shapes such as cube, arc, sphere, cone, ... and so on)

• Plans

5 to 7 are done.  So, I will plan to work on 8 and 9 first and then 1 and 2. Considering the time remaining for the summer, maybe I can finish 8 and 9, but not sure about 1 and 2. I will do my best and let's see how far we can go.

Weeks of 6/25/2014 - 7/8/2014

We just realized a significant issue in the system we have implemented so far.

• Current Implementation

 Currently, the 3D system in C-Snap works as follows:

1. Each object on the stage renders its image on its own (invisible) canvas
2. The objects are sorted in ascending order of Z-values
3. The objects are rendered on the stage (visible canvas) by the C-Snap drawing system

From the C-Snap drawing system perspective, 2D and 3D objects are more or less the same -- They are both SpriteMorphs that have an associated canvas each, and thus 2D and 3D objects can coexist without a problem.

However, this implementation has a significant limitation: we cannot render multiple objects that are both in front of and behind of one another. Instead, all we can do is change the rendering order of canvases. This issue is depicted in the following diagram:

• Solution to the Issue

One possible solution is to let three.js do the complicated 3D job. We can give three.js a large canvas (perhaps the one that StageMorph has) and let it render multiple 3D objects on that canvas. After that, we  somehow overlay multiple canvases for  2D objects and the single canvas for 3D object.

However, this solution is going to be a major change to the existing system and there will be a lot of technical challenges including the following:

1. Integration to the existing system
2. Overlapping between 2D and 3D objects
3. Drag and drop of 3D objects

Having said that, this solution is good in terms of the completeness of 3D support, so I would like to go ahead in this direction.

Weeks of 6/11/2014 - 6/24/2014

• Work plans

After talking to Charles, we agreed to implement the following 3D functinalities:

  * Transformation (move x y z)
  * Rotation (rotate n degrees in the x, m in y, c in z)
  * Scaling (make the thing .5 times it's current size in the z axis)
  * Textures (apply this image to all the surface on the object) 
 

• Transformation

Suppose we are looking at the stage on the X-Y plane from point A at (z_eye, 0), point C at (z_obj, h_obj) is projected as point B on the stage as follows:

Add caption

We can apply the same idea on the X-Axis too. So if we can implement this idea perfectly, then object movements on the Z-axis look naturally. However, what I have implemented so far is not perfect -- when moving an object in the Z direction, I only change the size of the object, but keep the x and y coordinates the same. This is illustrated in the following example:

The center of the object does not change and is always at (x, y) = (100, 100) -- logically and physically. In this example, the viewpoint is at (x, y, z) = (0, 0, 200), so following the idea presented above, the projected X and Y coordinates on the stage should be:

• For (x, y, z) = (100, 100, -100): (x_proj, y_proj) = (66.67, 66.67)
• For (x, y, z) = (100, 100, 100) : (x_proj, y_proj) = (200, 200) 

To implement this correctly, 3D objects need to have logical coordinates as well as physical coordinates on the stage, but it causes a major change to the existing 2D system. So, it might be less problematic if we keep the 3D system as it is currently implemented.

Week of 6/4/2014 - 6/10/2014

• 3D Costumes  

Last week, I demonstrated a few rotating 3D costumes, but they were working with specific scaling factors to show them on the stage nicely. To display arbitrary-sized 3D costumes on the stage, there are several options including the following:

1. Keep the original dimension of 3D costumes
2. Scale the dimension of 3D costumes to fit a specific-sized canvas

Currently, the size of the stage is (360, 480); however, available 3D objects do not always fit to this size. So, it seems to me that option 2 looks reasonable at this point. Moreover, we can change the size of 3D objects by the "change size by X" block.

I choose (160, 160) as the dimension of a canvas for 3D costumes on the stage.  To find appropriate scaling factor to fit this canvas, I used the idea described this Q & A page. As a result, a general 3D costume loading function is successfully implemented as follows:

In this figure, spheres are called bounding spheres and are displayed for debugging purpose. No matter how we rotate the 3D objects, they stay inside the spheres. I implemented these sphere to be scaled to fit the (160, 160) canvas.

• Issues
•  2D rotation still does not work properly. The size of the canvas gets bigger depending on the rotation.
•  Nesting for 3D operations are not implemented yet. 2D operations seem to work.

Week of 5/28/2014 - 6/3/2014

• 3D Costumes

I have been working on "real 3D costumes" this week. By "real 3D costumes", I mean you can load arbitrary 3D objects from three.js-compatible JSON files. Here are some examples of those JSON files: Walt Disney and Suzanne.

As suggested by Prof. Eglash, I added the following menu for loading 3D costumes:

Once you load 3D costumes from the newly added 3D Costumes menu, sprites on the stage look as follows:

• 3D Rotation (Turn)

3D Objects can be rotated around X, Y, and Z axes as depicted in the following figure.The Z-axis is not really visible on the stage, but logically it is there as if it is coming towards you.

To rotate objects in 3D, the following blocks are added this time. Names of the blocks are tentative (please let me know if you come up with a good one).

• 3D turn: This block is used to turn the object gradually. That is, it adds the specified parameters to the current object's degrees.

 

• 3D point: Unlike the 3D turn, this block rotate the object to point absolute degrees in the 3D space.

Here is an example of rotated objects:

• Github repository

The source code implementing these 3D functions can be found at:
  https://github.com/imais/Snap--Build-Your-Own-Blocks
Please note this is highly experimental, so use it at your own risk.

• Issues

• When rendering 3D objects, scaling factors are chosen to fit a 120 by 120 canvas. We need to figure out a way to properly scale and project 3D objects on the 2D stage.
• 2D rotation does not work properly. The size of the canvas gets bigger depending on the rotation.

Week of 5/21/2014 - 5/27/2014

• Experiments on 3D costumes

I have been focusing on implementing 3D costumes this week, and it works pretty well as shown in Figure 2.

Figure 1. Default 2D Costumes

Figure 2. 3D Costumes

Currently, 3D costumes can be loaded from the Import menu like other 2D costumes, and I am forcing the filename of a 3D costume to be "3d-[shape].[extension]" to identify the shape of the 3D costume. For example, the filename of a 3D cylinder has to be "3d-cylinder.png".

As for the color, a (perhaps randomly chosen) color for the default arrow shaped icon is taken over. We can see the colors for 2D costumes used in Figure 1 are also used in 3D costumes in Figure 2.

We can interchangeably select regular 2D and 3D costumes for a single sprite without a problem. Scripts can be written as well for 3D-costumed sprites.

• GUI for 3D shapes

Professor Eglash proposed the following three options for users to select 3D shapes:

 
  1) Each shape appears as an individual codelet:

  2) The shapes are in a drop down from one codelet:

  3) There is a list accessible from the file menu:

After a brief email discussion with the professor, Charles, and myself, we think it is the best if we have both option #1 for the Pen interface, and option #3 for the 3D costume selection interface.

• Issues 
• We cannot directly specify the diameter or length of 3D objects when rendering them using three.js because of the camera position. There may be a way to do it as discussed in this Q&A though.

• How to implement applications like VirtualWigwam ? To dynamically rotate a 3D space, we need to create a 3D stage that allows us to place multiple 3D objects in the same 3D space. That will introduce major design changes from the current C-snap. Perhaps, we should do the above option #1 and #3 on the existing 2D stage first and then implement a 3D stage as another mode ?

Week of 5/14/2014 - 5/20/2014

• Playing with Three.js

Just to get used to 3D rendering with Three.js, I followed the this instruction and successfully displayed a rotating 3D cube on Google Chrome.

• Displaying a 3D cube on the stage of C-Snap

Next step is to bring this 3D cube to the C-Snap's stage. As Prof. Eglash mentioned in an email, there seems to be two options to do that:

1. Pen
2. Costume

I thought option #1 seemed easier and the "stamp" function of C-Snap was a good start to learn how to display images on the stage. So, I started from the "doStamp" function, which displays an image of a Sprite, and tried to modified it to display a rotating 3D cube on the stage. Here is the summary of what I did:

• Added a new "stamp cube" codelet, which calls "doStampCube" function.
• Copied the "doStamp" function in object.js and named it "doStampCube", and then replaced context.drawImage() with three.js commands to draw a 3D cube. Also, added a step function to constantly render the cube for animation (thanks to suggestions from Charles & Libby!!).

Here is the result:

It looks good in a sense that the green 3D cube keeps rotating; however, as soon as you move the Sprite (the arrow-head shaped object), the animation stops as follows:

In this figure, the left half of the cube keeps rotating, but not the right half. This means that the area around the Sprite is subject to update, but not the other area. This is a reasonable behavior considering the fact that stamp is designed for still images.

At this point, I don't know if I should try to resolve this issue with option #1 or start working on option #2. I will keep looking both options for now.

Getting started

So, I will be working on a 3D-system for javascript-based C-Snap over the summer.
Perhaps the first goal would be to port the VirtualWigam application currently written in Java to C-Snap.

To Do:

1. Understand how C-Snap works 
2. Understand the requirements for the 3D-system in C-Snap
3. Learn Web-based 3D technologies in general and existing 3D libraries including three.js