How to validate a Mixi.jp signed OpenSocial request on App Engine

I know nothing about security, but had to validate a signed request and this experimentally seems to work.

Get the certificate from here:
http://developer.mixi.co.jp/appli/pc/lets_enjoy_making_mixiapp/require_servers

It is a long base64-encoded string that starts with -----BEGIN CERTIFICATE-----. Copy and paste it to base64 decoder. Store binary file "DecodedBase64.bin".

Run this command. This comes preinstalled on Mac Leopard.

openssl x509 -in DecodedBase64.bin -inform DER -noout -text

Copy and paste everything the hex values "Modulus (1024 bit):" to a text editor. Replace ":" and " " with "". Add \ to the end of each line. Now with this, you can use the instructions here to continue (the code after "which has been parsed, converted to hex value" on that page). Remember to put 0x in the beginning.

Using Google Spreadsheets with iGoogle to create a shareable dashboard for your startup

Google Analytics is excellent for tracking visits and can even be used for more advanced things like tracking conversions and events, but what if you want to track application-specific metrics which don't happen in a user's browser? For instance our Twitter meme aggregation site gets tweets using the Twitter API and stores them in our database, and the amount of tweets is something that I am interested in keeping an eye on. Here's what our dashboard looks like, and in this post I'll show you how you can effortlessly create your own.



Not only do the charts look pretty neat, but you can drag and drop them to rearrange them on the screen and even drag around inside each time series graph to see different spans of time. So where are these graphs coming from? A cronjob executes a simple Python script every quarter hour. The script queries our system for the load average, asks MySQL for the number of tweets and other data I want to chart. Each piece of data gets inserted into a separate Google Spreadsheet using the Google Spreadsheets API.

I can visit each of these spreadsheets and create charting gadgets (insert > gadget). Here I only used the time series gadget, which is like the one used on Google Finance, so you can easily zoom in to different periods of time and mouse over to see the absolute values. Clicking on the upper right corner of a gadget you'll find the option "add gadget to iGoogle". After you have added a few gadgets to your iGoogle, it may make sense to combine them into an iGoogle tab. Then on a tab's menu you'll find an option to "share this tab", and now any member of your team can view the data.

That's a summary of how this is done, but for the sake of completeness I will now mention some problems I encountered and share some code snippets so you won't necessarily have to go digging through the API.

First of all, one perhaps strange decision I made was to use a separate spreadsheet for each thing I track. One reason is that different data might be polled at different intervals, so the data is easier to handle this way as there won't be any blank fields for missing data. The more real reason however is a caveat I found, which is that each spreadsheet can contain no more than 200000 cells. There are around 35000 quarter hours in a year, so holding all data in the same spreadsheet would hit the limit in just a few months. I might also mention that I don't yet know how to get a charting gadget to automatically extend to cover all rows as the spreadsheet grows. If you know how to do that, please let me know and I'll be sure to mention the solution here.

Now some snippets. These use the Python library gdata-1.3.0. Logging in to Google Spreadsheets:


gd_client = gdata.spreadsheet.service.SpreadsheetsService()
gd_client.email = 'YOUR GOOGLE ACCOUNT EMAIL'
gd_client.password = 'YOUR GOOGLE ACCOUNT PASSWORD'
gd_client.source = 'exampleCo-exampleApp-1'
gd_client.ProgrammaticLogin()


Below is the snippet for inserting one row to a certain spreadsheet. Each spreadsheet can consist of several worksheets. Not sure if it's always true, but experimentally I found that the first sheet seems to be called 'od6'. The spreadsheet_id you can find by creating a spreadsheet, then looking at your address bar and copy the base64 looking string in the "key" GET argument. The value here is any piece of data that you want to track, for example the number for your current load average.


def newValue(spreadsheet_id, value):
worksheet_id = 'od6'
dict = {'datetime':time.strftime("%m/%d/%Y %H:%M:%S"), 'value':value}
gd_client.InsertRow(dict, spreadsheet_id, worksheet_id)


Finally here is my complete list of things I import in the beginning of my script. It contains many unnecessary includes for these particular snippets, but I thought it might still prove helpful if you are trying to get the code to work.


try:
from xml.etree import ElementTree
except ImportError:
from elementtree import ElementTree
import gdata.spreadsheet.service
import gdata.service
import atom.service
import gdata.spreadsheet
import atom
import getpass
import string
import time
import urllib
import os
import re


Thanks for reading and please comment on Hacker News (preferred) or here in the comments. I'm interested in becoming a better blogger too, so any suggestions on improving my writing would be appreciated as well.

iPhone dev 20

Found one piece of the puzzle. To add views so that they are visible, the window has a method called "addSubview", seems to be similar to addChild in ActionScript. A default windowed iPhone project already has a "window" attribute in the delegate class. applicationDidFinishLaunching also contains a reference to the XIB file. A view controller has an initWithNibName method that takes the name of a XIB file.

Found an answer to the source code generation question too:

It doesn’t generate source code, instead it allows you to manipulate objects directly and then save those objects in an archive called a nib file. At runtime, when a nib file is loaded the objects are unarchived and restored to the state they were in when you saved the file

iPhone dev 19

So this is how the code knows about objects from Interface Builder. The code has a class called (something)AppDelegate, which has an applicationDidFinishLaunching method that gets called at start. The *AppDelegate class can have IBOutlets for different things like buttons and window from IB, so that it can reference them.

But still, how do I add a new XIB file containing a different view to the project and reference it from code?

iPhone dev 18

I'm back from the sidetrack, at least for now. Game prototypes and MySpace widgets were taking up my time, and probably a bit later will again, but today I'm figuring out iPhone stuff again.

I really still don't understand the relationship between the Interface Builder and code. If I make a view in Interface Builder and add the XIB file to my project and build the project, did it now create a class definition for it automatically? How can I even find out? I feel frustrated that magical things are happening in the background, code is apparently created that I have no awareness of. But this feeling is familiar, I know it will be replaced by understanding if I just continue on this long enough. Things will click into place, and then it will be a breeze to create things. That'll keep me motivated to continue on.

What I want to accomplish is simple. I want a button in the main screen, that when pressed slides out another view which is also constructed in the Interface Builder. But to do this, somehow I need to reference this other view from code, and I don't know how it appears there. I see at least these options of how this might work:

a) Perhaps if I just add a XIB file to a project, Xcode does something magical behind the scenes that results in a class definition being available to all code, which I then have to instantiate. This seems unlikely somehow.

b) Perhaps adding a XIB file to a project causes objects defined in the file to be instantiated. This seems likely, I remember reading that XIB files are kind of frozen object relationships. But when they are thawed, where are they?

c) Adding a XIB file to a project is not enough by itself, but you need to create your own proxy classes in Xcode to be able to reference them. But this wouldn't solve the issue previously mentioned. Even if I know the class names, I still need to reference the thawed objects.

Gimme the thawed objects!

YouTube is easy?

I read a lot of blogs where developers voice their opinions on how to do successful things. Often YouTube is raised as an example of a service that was easy to implement, but filled a real need in a user-friendly way (unlike Google Video) and so became popular. I agree with everything except the "easy" part. If I think about web services, YouTube really doesn't seem comparatively on the easy side.

YouTube seems like a total nightmare to scale for starters. Their bandwidth usage boggles the mind. YouTube currently uses more bandwidth than the entire Internet did less than eight years ago! They spend an estimated million bucks PER DAY on bandwidth.

So OK, perhaps when they say it was easy to implement, they mean when it started, not the way it is right now. Some blogs like to point out that because they use PHP and MySQL, this makes it easy to implement. Granted, the site UI part itself doesn't seem to be especially difficult. But is that really the meaty part of YouTube? Personally I think transcoding is the essential and also the difficult part. Perhaps they use something like FFMPEG to do that, making it easier than I imagine, but they really had to overcome some kind of a mental hurdle to realize that this uploading thing could ever work well enough.

Also from a legal standpoint it does not seem easy. YouTube seems like it might have been much easier to start from China or somewhere with less strict copyright laws. Being a company in the US, I wonder how they got any sleep at night with lawsuits and cease and desists flying at them left and right.

And now they have yet another tiny little detail: how to even make any money from all this. The founders Chad Hurley and Steve Chen of course don't really have to worry too much about this any longer, but I'm thinking about this from the point of view of the business itself. Assuming they really are spending $1M/day on bandwidth, and believing the compete.com estimate that they have 64M monthly uniques, they would need to extract about 50 cents per user per month just to cover the bandwidth cost.

I don't believe they are doomed. It can likely all be made to work, maybe that $1M/day bandwidth will cost a small fraction of that after 10 years. Maybe there is some magical d'uh-obvious moneymaking scheme they will come up with. What I just mean is that none of this makes me feel like YouTube should be the first company people pick as an example of something "easy" to do.

Here's how to monetize Twitter

Add a gifting feature called "bribes", similar to the gifts you can send on Facebook, except they are gifts that all of your followers will get. They are collectible items totally controlled by Twitter, and cost real money to buy. You would buy these items to reward your followers and to encourage more people to follow you.

Never let a Finnish man drink too much

A drunken 38 year old man caused extensive damages and kept the police occupied, going on a mad nightly rampage. All of this happened in the span of a few hours.

To start the night, he stole a woman's wallet in a bar. After the bar closed, he started walking home, but stopped on the way to kick down a dozen gravestones at the graveyard. After that he broke some windows of nearby houses, and lit a moped on fire. Soon after this he also lit three garbage collection points on fire. One of the fires spread to a van parked next to it.

Next he wandered to a kindergarten and lit on fire a baby carriage left outside. Then he went on to a deaf peoples' home, attempting to conceal his identity by putting a shopping bag on his head. There he beat a night shift worker with a stick, broke windows and equipment, then demanding to get medicine. A female nurse managed to lock herself in a room and alert the police by phone. However an ambulance arrived before the police did, which the man then stole and drove around the town for half an hour, before smashing the windows and abandoning it. After this he called a taxi and went home to sleep.

The police already had an idea who the culprit might be, so they catched him from his home after he had already fallen asleep. He was interrogated and blames alcohol for his behavior.

( this was a translation of a 1994 news clipping from Finland )

Protostorming again: Build a School

In this MySpace game you maintain your own school. You and your friends take the roles of teachers, competing against other schools for the highest GPA. GPA for a school might be calculated as (your study points)/(best player's study points)*5. The teachers collect "study points" for a school by giving the students lectures in a simple quiz minigame (prototype screenshot below).



Reaching certain study point thresholds would cause events to occur, which might lead to positive or negative consequences, or cause the player to receive an item. Players would often receive the same items, and would be then prompted for example "you don't need two blackboards! how about giving this to a friend?". This would be the second viral channel after teacher recruiting.

Third one could be a minigame where you have to gather friends to upgrade a school. At first the school would be just outside on the grass. Secondly it could have walls, and finally a nice floor too. How this upgrading part would work, I'm not sure. Perhaps it could even just be an event when a certain study point limit is reached, although then it wouldn't help the virality (but would still reward players to continue playing). (Lil) Green Patch is basically just about giving items, and has thrived on just that. I tend to think in too complex terms. Maybe minigames aren't even necessary, or might even complicate things too much? Perhaps it suffices to incentivize the players by donating all or part of revenues towards building actual schools in 3rd world countries, which the best players could name.

Seth Godin in my head

Say nothing, but say it big, inspire. No facts, no proof, but it tastes so sweet. Seth Godin saying in my head "be a leader, you must do it, the world needs you!". How can you not smile, even start to believe? Then, a month from now I might not be any different for having listened to his audio book, but for that short while I felt great. Donuts for the mind.

Please smash head against keyboard

Fire game prototype

I can survive 163 seconds, how about you? I learned from making this prototype that having a good difficulty curve is important. In this prototype, the player is entertained for the first 5 seconds, then bored 60 seconds, then perhaps slightly curious for the next 30 seconds and finally faced with an abruptly impossible difficulty level, making the player feel they died due to no fault of their own -> not fun.

Rotoscoping experiment

I did a small experiment to see how difficult it would be to track joints from video material. I was meaning to do this before back in the DOS days after seeing the cool rotoscoping in Another World (Out of this World), but back then it was too difficult to get the video into some form that I could easily read from code.

What I wanted was dancing material with an unchanging background and a static camera. Now there is YouTube so it is really easy to find this. Next I downloaded the FLV movie using a 3rd party tool (there are many). I used "FLV Extract" to get the video stream from the FLV file as an AVI file. Then I used Virtualdub to crop it to a short segment. Converted that segment into a GIF animation with GIF Movie Gear, then finally imported the animation to a Flash timeline.


Next I attempted to track the left wrist of the dancer to see how much work it is. As a first test it took about 5 minutes of concentrated effort to mark 50 frames. Mostly I was so slow because I needed to move forward and create keyframes in the timeline and drag the joint marker with my mouse. I think if I made a small tool that let me just tap on a joint with my drawing pad, I could manage in under 5 seconds per joint per frame.

At least 20 joints would be necessary to make a stick figure dance like the dancer. Three minutes of YouTube video at 12 fps is 2160 frames times 20 joints is 43200 taps times 5 seconds is 60 hours! And of course this data is 2D, so it isn't even clear what could be accomplished with it. One thing did occur to me though -> you could use this as an affordable motion capture solution for games. Suppose you have a front camera and a side camera filming the same footage, so 120 hours of work to track it. Send the task to China, suppose it costs $5/hour -> total $600 for probably all of the motions needed for a small game.

Shipyard

Rough prototype of the day. The graphics are placeholders I found with Google image search. To try this, select a tile from "power", then click on a grid to place some. That's all you can do for now.



I've played many space shooter games, but a thing that bothers me is that the ships never have any kind of internal structure. Your ship tends to be just a scalar value like HP, and when it reaches 0 you die. I want ships to have inner structure. Imagine a space battle. Wouldn't you much prefer to hit your enemy in their warp core to cause a devastating chain reaction that destroys the entire ship?

Social networks as maps of real connections

When thinking of what kind of apps to make for social networks, thinking about what people do in their everyday life can be a great source of ideas. A big part of social networks is mapping out real connections. The concept of "Social Graph" is so well-known as to become even a bit nauseating to think about one bit more, but there are other connections besides just friendships, not all of which are necessarily mapped out yet. For some, it probably makes no sense to map them.

A person has...
a name (Name Analyzer -- at least I thought about this one!) both official and nicknames (Nickname)
age (Birthday Calendar -- still kicking myself for not thinking about this first)
x,y,z location at each time they existed -> birthplace, countries they visited (Places I've Been), grave
photons disturbed by them (Photos/posting of videos)
likes/dislikes about food, movies, music, smells, colors...
personality (Compare People)
experiences
knowledge (Who Has The Biggest Brain, Scrabble, Geo Challenge)
degrees/certificates
medical conditions, sleep cycle, height, weight, dna (a bit too costly to compare your DNA with friends, but we'll get there ;) )
ownership of physical/virtual items (can own a car, a house, stocks, money, music recordings, WoW gold...)
relationships with humans -- idols, friendships, relatives, enemies, employees/employers
pets
opinions about...
- other people (Hot or Not, Compare People)
- what is wrong and right
- politics
- which gods if any exist
...

"Groups" by Facebook is very much a meta-application in that almost anything can be expressed as a group. It may be that applications that can add value beyond what a membership to a group could offer will be successful. One other app I'm kicking myself for not coming up with is Bumper Sticker, which solves the meta-problem that groups solves, but in a more visual way. If there is any other uninvented angle than Groups and Bumper Sticker to this, it will be huge. Other thing to think about is whether there would be value in aggregating the connections of your friends in some way. For example you have an age, and while that in itself might make a great app (Brain Age), aggregating friends' ages also gives one (Birthday Calendar).

If you can think of any other general sources of ideas, please post some. I'm sure there are still a few d'uh-obvious ideas out there to be discovered!

Secant square

Coded something just for fun for a change.



This is a square drawing routine that uses trigonometry instead of the usual way to create a square. The sliders skew the trig equation in different ways.

Harry the load balancer - a true story

Harry works at an ISP called Elisa, in a depressing country near the north pole called "Finland". Imagine yourself as Harry. Surrounded by the cold, dark reality that is Finland, sitting in a noisy computer room, doing possibly the most menial job on the planet. Harry is a load balancer. He gets requests, then just randomly forwards them to one of his clients. Even mail sorters have more fun. "And probably wilder parties too", Harry complained to his friend Tom, also a load balancer. "And probably wilder parties too", Tom said with his identical software. Their shared conclusion was to stop putting up with this situation any longer. But what can you do, how can you protest if you are but a tiny load balancer?

Tom and Harry pondered this in parallel. They could of course call it quits completely, or commit suicide as a Finnish load balancer might do, but Harry had other plans. "Let's have some fun while we're still around", Harry and Tom decided simultaneously. The real nature of Harry's job was to split DHCP requests coming from the 500,000 customers of the ISP. Tom wasn't currently connected. His job was to act as the back-up guy to Harry, in case anything happened to him. Well, now something was about to happen to him. "Wouldn't it be fun to see what happens if I forward all of the traffic to a single client?", Harry thought full of jubilant delight. "Don't do it, man", the DHCP server pleaded, but it was too late. The DHCP server exploded violently, leaving a bloody mess on the server room floor.

Harry should have been more careful with his crime. There was a rat in the room, not the animal kind of rat, but the kind of rat that rats out mischievous load balancers. That rat was a box configured to send an SMS to a human in case anything like this would happen. "Get ready to sing Bicycle Built for Two", the rat said as human steps could be heard walking at a steady pace towards the computer room. The human, let's call him Dave, although that certainly isn't his name, opened the door and entered the computer room. "I'm afraid", Harry whimpered as the human started to unplug his wires. "But, good my brother, do not, as some ungracious pastors do. Show me the steep and thorny way to heaven whilst like a puffed and reckless libertine himself the primrose path of dalliance treads and wrecks not his own.", Harry might have said to Tom at this point if he was into Shakespeare, but instead he said: "brother, your time to serve has come". Harry was unplugged now, with Tom in his place. Dave was satisfied, and decided to walk to the soda machine to get a cola.

What happened while Dave was away should be clear. Tom being identical to Harry in every way, had also decided to be naughty today, so all the load was still being un-balanced to a single agonized DHCP server. However it was getting quite late at this point, so the traffic wasn't that much any longer. So superficially, everything was fine, and Dave could go home to watch the full DVD release of My Little Pony or whatever it is that IT workers do at home. Next morning of course would be a bloodbath.

Based on a true story on the cause of 500,000 Finnish Internet users -- or 10% of the Finnish population -- losing their Internet connections for hours. Original Finnish language news article.

Where to live?

I've been trying to find a place to live where my savings would last for a long time. I found this site: Cheapest Countries in Asia. It says Indonesia is the cheapest. I believe this list, because at least Thailand definitely really is three times cheaper than Japan.

So now I have a list of cheap places. The next piece in this puzzle would be to discover which places have good Internet connectivity. Here are SpeedTest results for Indonesia. That shows some places do have nice speeds, but it says nothing about reliability. Doesn't really matter if it's broadband speed, if it only works every other day.

The search continues.

App got acquired

A small application I made was acquired by a company yesterday. I think the price was fair, this deal is very likely a win-win for both me and the new owner of the app. The price was a multiple lower than what I would have asked just a few months ago. While page impressions have remained stable, CPM -- basically how much money you can get from banner ads -- was much lower.

It may be so that this was not a good deal for me, because if I had kept the app it could have continued to make ad revenue for the rest of my life. It's very difficult to say, but at least I now certainly have enough to live on for a few years. Also maintaining a simple app like that for years would certainly have distracted me, now I have space in my mind to come up with other things. I'm always bursting with ideas, but often too distracted to act on them.

It's good to sometimes know to let go.

Another World nostalgia gun

Nowadays, I mostly code out of greed. Having some success getting some ad revenue, I try to figure out ways to get moar. However there are some things which I would like to create just for fun. One of them is adding the Another World gun to the game Half-Life 2 Deathmatch. The illustration below of how the gun works is lifted from the author Eric Chahi's page for the Another World game.



Another World (called "Out of this World" in USA) was one of the first, if not the first 2D computer game to use polygonal graphics. It was also remarkable in that the very graphical game was drawn and programmed by the author alone, outside help was only used for the music. It is rare to have both artistic and programming talent in the same person, although that may be somewhat a self-fulfilling prophecy as because of this belief programmers might get a bit discouraged from even trying. Well, in any case the gun works such that pressing quickly fires a thin ray, holding button down for a moment and then depressing creates a shield, and holding for a long time and depressing creates a huge burst which can also destroy shields. Shields can be walked through, but thin rays cannot pass them. This enables you to extend an arm from behind the shield to shoot from safety.



I imagine in 3D this would work very similarly, except of course the shield would be a plane, extending a limited amount. For artwork it could use the shields already in the game, as seen in the picture above. So what's preventing me from adding this? Well, since it's a gun I think it could not be just a server modification, and it seems like overkill to release a client-side mod just to add one gun to it. If I did that, well likely nobody would install it, and nothing is sadder than a nice mod nobody uses. I've seen some great mods that added really nice weaponry like flame throwers, but nobody was playing those mods. Now that leads me to the next suggestion.

Multiplayer deathmatch games like this should have a plug-in system, where servers could push sandboxed client-side code snippets to the players. This would create an incentive for server owners to create new stuff and weapons, since it would attract more players to their servers. It would also be good for the players, because then we'd have all that new stuff. Sort of like Second Life, except with an engine more suitable for combat. The controls and "gameplay" feel in Second Life are horrible, so it certainly doesn't fill this need.

EOF.