很多大规模机器学习系统（和其他大规模分布式计算系统）都需要有自动重启的功能。很多大公司有自己的分布式操作系统可以负责自动重启失败的任务（tasks）。可以如果没有这样的分布式操作系统怎么办？daemon命令可以是一个“穷人“的的解法。

启动一个任务

wangyi@localhost:~$ps daemon --name=data_server_1 --respawn -- sleep 999

启动一个任务，名字是data_server_1（对应/tmp/data_server_1.pid这个文件；这个文件用来保证不能再运行同一个daemon命令来启动同一个任务），这个任务执行命令 sleep 999。其中–respawn参数的意思是：如果 — 之后的命令（任务）挂了，daemon应该自动重启之。

此时，启动了一个daemon进程，它会fork（spawn）出来一个进程来执行命令sleep 999:

wangyi@localhost:~$ps auxwww | grep sleep
wangyi 15129 0.0 0.0 2432748 472 ?? S 10:47AM 0:00.00 sleep 999
wangyi 15128 0.0 0.0 2432896 312 ?? Ss 10:47AM 0:00.00 daemon --name=data_server_1 --respawn -- sleep 999

另外，/tmp/data_server_1.pid里存放的是daemon命令得到的process id：

wangyi@localhost:~$cat /tmp/data_server_1.pid
15128

任务的自动重启：

如果我们杀死一个由daemon fork的进程：

wangyi@localhost: kill 15129

Fork这个进程的daemon 进程会自动重启这个任务：

wangyi@localhost:~$ps auxwww | grep sleep
wangyi 15147 0.0 0.0 2432748 472 ?? S 10:47AM 0:00.00 sleep 999
wangyi 15128 0.0 0.0 2432896 312 ?? Ss 10:47AM 0:00.00 daemon --name=data_server_1 --respawn -- sleep 999

可以看到，任务15129虽然被杀死了，但是一个新的任务 15147 被启动来执行sleep 999

再启动一个任务

可以启动另一个任务，只要名字和之前的不同，执行的命令可以是同一个：

wangyi@localhost:~$daemon --name=data_server_2 --respawn -- sleep 999

wangyi@localhost:~$ps auxwww | grep sleep
wangyi 15201 0.0 0.0 2432748 472 ?? S 10:52AM 0:00.00 sleep 999
wangyi 15200 0.0 0.0 2432896 312 ?? Ss 10:52AM 0:00.00 daemon --name=data_server_2 --respawn -- sleep 999
wangyi 15129 0.0 0.0 2432748 472 ?? S 10:47AM 0:00.00 sleep 999
wangyi 15128 0.0 0.0 2432896 312 ?? Ss 10:47AM 0:00.00 daemon --name=data_server_1 --respawn -- sleep 999

这样我们就会有两个pid文件：

wangyi@localhost:~$ls /tmp/data_server*
/tmp/data_server_1.pid /tmp/data_server_2.pid

wangyi@localhost:~$cat /tmp/data_server*
15128
15200

停止任务：

用命令行参数–stop和–name来杀死任务：

wangyi@localhost:~$daemon --stop --name=data_server_1
wangyi@localhost:~$daemon --stop --name=data_server_2
wangyi@localhost:~$ps auxwww | grep sleep

One of the most interesting points in this paper is the asynchronous SGD algorithm, which enables a parallel (distributed) software architecture that is scalable and can make use of thousands CPUs.

To apply SGD to large data sets, we introduce Downpour SGD, a variant of asynchronous stochastic gradient descent that uses multiple replicas of a single DistBelief model. The basic approach is as follows: We divide the training data into a number of subsets and run a copy of the model on each of these subsets. The models communicate updates through a centralized parameter server, which keeps the current state of all parameters for the model, sharded across many machines (e.g., if we have 10 parameter server shards, each shard is responsible for storing and applying updates to 1/10th of the model parameters) (Figure 2). This approach is asynchronous in two distinct aspects: the model replicas run independently of each other, and the parameter server shards also run independently of one another.

Intuitively, the asynchronous algorithm looks like a hack, or a compromise between the effectiveness of the mathematical algorithm and the scalability of the distributed system. But to our surprise, the authors claimed that the asynchronous algorithm works more effective than synchronous SGD.

Why??

My understand is that traditional gradient-based optimization is like a bee flying along the direction defined by the current gradient. In batch learning, the direction is computed using the whole training data set. In SGD, the direction is computed using a randomly selected mini-batch of the data.

In contrast, the asynchronous parallel SGD works like a swamp of bees, each flies along a distinct direction. These directions vary because they are computed from the asynchronously updated parameters at the beginning of each mini-batch. For the same reason, these bees wouldn’t be far away.

The swamp of bees optimize collaboratively and covers a region like region-based optimization, where the region is composed of a set of points. This, I think, is the reason that parallel asynchronous SGD works better than traditional gradient-base optimization algorithms.

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>

  <property>
    <name>hadoop.tmp.dir</name>
    <value>/Users/${user.name}/hdfs-${user.name}</value>
    <final>true</final>
  </property>
</configuration>

Say, that a binary A uses some, but not all, symbols in a static library B, the linker would extract those symbol in-use from B and put them into A.

But sometimes, we do not want the linker to be as smart as this; instead, we might want all symbols in B are put into A.

To do so using GNU ln (usually comes with Linux), we say

g++ -o A -Wl,-whole-archive libB.a -Wl,-no-whole-archive A.cc

To do exactly the same thing using ln from Mac OS X, we say

g++ -o A A.cc -all_load libB.a

A typical situation that depends on above tricks is to link a self-registering class factory, and here is an example.

Cloning a Subversion Repository

The first step is to actually clone a Subversion repository using Git. This is accomplished by executing the ‘git svn clone’:

$ git svn clone <subversion-url> ?optional-directory-name?

For example,

$ git svn clone file:///Users/blair/git-svn/SVNROOT git_repo
Initialized empty Git repository in /Users/blair/git-svn/git_repo/.git/
r1 = e1e2a18c6537614f2bebc5c0dfb4ac6f8c829a3b (git-svn)
	A	file1.c
	A	Makefile
r2 = 58e044c16678b0a8b92e4dbed8f0a69d48ba9ec1 (git-svn)
Checked out HEAD:
  file:///Users/blair/git-svn/SVNROOT/trunk r2

As you can see, this creates a new Git repository in the directory named ‘MyGitRepository’, populates the Git history with all Subversion commits to the trunk, and checks out the latest HEAD.

Note that this operation will generally take longer than a corresponding ‘git clone’ command, as the entire history for the Subversion repository in question must be downloaded commit by commit. For large projects, this process can take hours. Thus it’s best to execute this command only when you have some time to kill.

Synchronizing With Upstream

When using Subversion, you synchronize with upstream changes by running ‘svn update’, which pulls down all of the changes made to your Subversion branch and merges your working copy with those changes.

With git-svn, you accomplish a similar task via the ‘git svn rebase’ command.

bucky:git_repo blair$ git svn rebase
	M	Makefile
r3 = ae409c2f5fe0831f22d6dc891652b5f9159f35de (git-svn)
First, rewinding head to replay your work on top of it...
Applying: Local git commit.

So what does this tell us? Well, first, you see a list of modified files. The next line informs us that revision 3 from Subversion is being stored as commit ae409c2 in our local Git repository. No problems thus far. But, what about that line informing us Git is rewinding and replaying our work? Well, this is where ‘svn update’ and ‘git svn rebase’ differ slightly.

Remember that, with a Subversion repository, you don’t have the concept of offline commits. So when ‘svn update’ executes, all it does is pull down all of the commits from the Subversion repository, and replay the changes in your working copy.

Under Git, you’ve likely been making a series of changes and committing them in nice small pieces (you are making regular commits, aren’t you?). So, when ‘git svn rebase’ is executed, it rolls back all of your local commits, pulls down all of the commits from Subversion, and then reapplies your local commits as if you had made them on the current HEAD of the Subversion branch. This has much the same logical effect as ‘svn update’ (all of your local changes are applied on top of the current Subversion HEAD), but may appear a little odd at first glance.

Just like ‘svn update’, it’s a good idea to execute ‘git svn rebase’ periodically in order to limit the number of conflicts you encounter when it comes time to integrate your changes back into the main repository.

Pushing Changes Upstream

So you’ve been hacking away in your Git repository and now have a series of local commits that need to be pushed back to the main Subversion repository. The first step is to execute ‘git svn rebase’ and make sure your local Git repository is up-to-date with the Subversion repository. Next, use the command ‘git svn dcommit’ to push all of your local Git commits back to the Subversion repository.

bucky:git_repo blair$ git svn dcommit
Committing to file:///Users/blair/git-svn/SVNROOT/trunk ...
	M	file1.c
Committed r4
	M	file1.c
r4 = 31a4b40b05e1b42f34dd22c34936f43dd5be90ec (git-svn)
No changes between current HEAD and refs/remotes/git-svn
Resetting to the latest refs/remotes/git-svn

Now all of your local Git commits are present in the Subversion repository and you can continue hacking away locally.

Rules and Guidelines

There are a few rules you should follow when using Git as a Subversion client. Many of these are due to the fact that Git is a much more capable system than Subversion and some of the Git features simply do not work in Subversion.

• Do not dcommit Git merge commits to the Subversion repository. Subversion doesn’t handle merges in the same way as Git, and this will cause problems. This means you should keep your Git development history linear (i.e., no merging from other branches, just rebasing).
• Do not amend, reorder, or otherwise change commits that have been dcommited to Subversion. This is essentially the same rule as not changing Git commits that have been pushed to public repositories. Subversion cannot handle modifying or reordering commits.

Short version:

• Merge takes all the changes in one branch and merge them into another branch in one commit.
• Rebase says I want the point at which I branched to move to a new starting point

So when do you use either one?

Merge

• Let’s say you have created a branch for the purpose of developing a single feature. When you want to bring those changes back to master, you probably want merge (you don’t care about maintaining all of the interim commits).

Rebase

• A second scenario would be if you started doing some development and then another developer made an unrelated change. You probably want to pull and then rebase to base your changes from the current version from the repo.

Facebook广告（非mobile device广告）展示在电脑屏幕的右边。实际上广告即内容，或者说ads就是sponsored content。如果自己做了content（比如页面、相册之类），要sponsor一下是很容易的。这里就不多说广告投放的体验了。

以上是背景。以下是这种背景下我的体验：

1. 一开始很多page view里压根儿没有广告。后来出了一个为“外籍人士推荐床垫”的广告：
   
   注意，这里因为我用了加拿大的vpn，Facebook好像把我识别成来西方世界的海外人士了。这里有“地域定向”feature。
   随后又出了一个”外籍人士在美国报税的代理“的广告。把我当做”移民“，（可能是因为我很久之前用这个账号登陆Facebook时是在国内，那时它还没被封），这说明地域feature和时间feature是有某种cross的。
2. 点击了上面床垫广告之后，接下来几屏幕都是满满的广告。这里有”广告点击欲望识别“feature。
   而那个点过的床垫广告，总是出现在靠前的位置，比如：
   
   说明有情境广告点击重定向。这个挺好，既然我对那个床垫感兴趣，多出几次避免我找不着。
3. 随后，我在Facebook的搜索功能里搜了一个“App推广”，之后给我出的广告里就多了一项“海外游戏推广”，而且这个广告多次刷屏之后一直都在靠前的位置。这不就是搜索广告重定向吗？
   
   注意：不同Facebook页面里，右侧广告位宽度不同；而Facebook会自动重排列同样的creatives，以适应不同广告位的宽度。
4. 最后，广告即内容（请注意下面抓图中最后一个不是广告而是photo album；上面抓图里也有events）：
   

~/usr/bin/gdb -i=mi hello_world

What I want is to use bridge as a proxy, so when I am working on the development workstation (dev) in the lab, I can access the Internet, including Github, Google and other services.

Due to the firewall restriction, bridge along cannot be a full-fledged proxy; but I am lucky to have a VPS (free) in the Internet. So what I really need is to set up a tunnel from bridge to free using dynamic forwarding:

bridge$ ssh -D 7777 `free` -p 80

Note that since bridge can only access services listening on port 80, we must configure SSHD on free to listen on 80.

However, free is a Web server and the Nginx running on it already takes port 80. So we need to multiplex port 80 for both Nginx and SSD on free. After Googling around, I found the magical program SSLH (http://www.rutschle.net/tech/sslh.shtml).

SSLH is a light port-de-multiplexing server in 2000 lines of C code. It listens on specified port, and forward connections to the port to other services by testing on the first data packet and judging the protocol of the connection.

Here follows how I use SSLH to multiplex port 80 on free.

First, I download and build SSLH. On bridge, issue the following commands:

wget http://www.rutschle.net/tech/sslh-1.14.tar.gz
tar xzf sslh-1.14.tar.gz
cd sslh
make

Building sshlh requires libconfig and libwrap. To install them on Ubuntu systems, use:

sudo apt-get install libwrap0-dev libconfig8-dev

To make sure how to install them on Debian or other systems, please refer to the README file distributed with SSLH code.

Once successfully built, we would have two binary files: sslh-fork and sslh-select. The former forwards by forking processes, and is thus slow; whereas the latter forwards by using select, which is much more efficient. However, sslh-fork had been tested in many use cases and is guaranteed robust, whereas sslh-select requires more tests. Since free runs a simple Web service, I just sslh-fork:

sudo cp sslh-fork /usr/sbin

I also copied the default configuration file and start shell script:

sudo cp ./scripts/etc.default.sslh /etc/default/sslh
sudo cp scripts/etc.init.d.sslh /etc/init.d/sslh

Then edit /etc/default/sslh to be as follows:

LISTEN=0.0.0.0:80
HTTP=localhost:8080
SSH=localhost:22
SSL=localhost:443
USER=nobody
PID=/var/run/sslh.pid

The first line makes SSLH listens on port 80 of all network interfaces. The second line specifies the server (Nginx) to which HTTP connections are forwarded. The third line specifies the SSHD.

To enable above configuration, I also need to edit /etc/init.d/sslh, to ensure that the SSLH will be started by the following command line:

$DAEMON --user ${USER} --pidfile ${PID} --listen ${LISTEN} --ssh ${SSH} --ssl ${SSL} --http ${HTTP}

Note that variables LISTEN, SSH, SSL, and HTTP are read from above configuration file.

Since the SSHD on free had already been configured to listen on port 22, I do nothing to it. But I need to make Nginx listens on localhost:8080 (port 8080 on the loop back interface) instead of 0.0.0.0:80 (port 80 on all network interfaces).

My Nginx is installed in /usr/local/nginx and configuration files are in /usr/local/nginx/conf. What I did is to ensure that all virtual servers defined in /usr/local/nginx/conf/nginx.conf and other config files included by nginx.conf listen on localhost:8080. A fast command is:

sudo sed -i'' 's/listen *80/listen localhost:8080/' *.conf

Then we restart Nginx, and make sure that no service is listening on port 80 by using the following command:

netstat -tpln | grep 80

At this moment, it is correct if we cannot access the Web service served by Nginx from any computer other than localhost. (Because we edited the nginx.conf to bind all Nginx virtual servers to the loop back interface, localhost.)

Then we start the SSLH service:

sudo /etc/init.d/sslh start

Everything is done!

If everything is ok, we should be able to connect to free via SSH and port 80:

ssh free -p 80

We should also be able to access Nginx server on free from a Web browser with URL http://free.