Link All Symbols in a Library into Binaries

Morden linkers are smart. But sometimes too smart than what we need, that we would have to tell them what exactly we want.

Say, that a binary A uses some, but not all, symbols in a static library B, the linker would extract those symbol in-use from B and put them into A.

But sometimes, we do not want the linker to be as smart as this; instead, we might want all symbols in B are put into A.

To do so using GNU ln (usually comes with Linux), we say

g++ -o A -Wl,-whole-archive libB.a -Wl,-no-whole-archive A.cc

To do exactly the same thing using ln from Mac OS X, we say

g++ -o A A.cc -all_load libB.a

A typical situation that depends on above tricks is to link a self-registering class factory, and here is an example.

Go Compiler Is Producing Faster Code

According to Fig.8 in this paper published in 2011, given the Havlack loop recognition algorithm implemented using Go and C++, the running time of the Go version is 7 times of that of the C++ version. Even after tuning the Go version, the ratio is still 5.5x.

This gap seems shrinks since the paper was published.  Today, I ran the same program on my macbook with OS X Mountain Lion, I found the ratios are 3.11 and 2.18 respectively.  I used Go 1.0.3 and i686-apple-darwin11-llvm-g++-4.2 (GCC) 4.2.1.

Why Go? Use Racket!

Go	Racket	C/C++
channel	channel	blocking queue
goroutine	thread	light-weighted thread
select	sync	epoll/kqueue

I had expected Go could save us from the heavy burden of programming complex network applications (Web servers at frontend and RPC server at backend). However, I am disappointed.

It is true that Go’s concurrent programming facilities (goroutine, channel and select) works great and can remove all the asynchronous RPC calls and chains of callbacks.  For how Go can do this, you might want to watch Rob Pike’s talk, Go Concurrency Patterns, on Google I/O 2012.

However, the type system of Go is so complex; programmers need to know all the details about the implementation of standard types, before they can use them correctly.  For example, it is efficient to pass a channel as the parameter of a function all, because a channel in Go is as simple as a pointer to the channel data structure implemented in C.  It is the same for map and some other types.  But it is inefficient to pass an array or struct; instead, we should pass pointers to these types.

Another example is about interface, the base of polymorphism in Go. The reflection mechanism of Go makes it convenient to judge the real type referred to by an interface variable. But we are not allowed to convert the interface into that type. Instead, we can convert it into a pointer to that type.  This is because an interface variable is a pair of two pointers, one points to the real variable, and the other one points to a method list.

Given the pros and cons of Go, what I want is a language, which has a comprehensive type system and implements the CSP (communicating sequential processes) model. Also, I hope this language is open source (as Go), has a Web based third party libraries creation and publication system, and can express complex computing logic in an intuitive way.  I am feeling lucky that I found one — Racket.  Racket was formerly known as PLT Scheme.  I used it to learn Lisp programming 8 years ago.  I won’t know it was renamed if my old friend, Yin, do not tell me.

Racket is open source. It is an enhanced version of Scheme with macro and pattern matching functions. So it is much more powerful than C/C++/Java/Go in expressing complex algorithms. For example, using C/C++/Java, a multi-page Web-based wizard would correspond to a set of HTTP handlers, each for a page. However, using continuation, a well-known programming construct in Lisp, we would need only one handler! (Here is an example.)

Racket has hundreds of third party libraries, including network programming and Google protobuf, published on PLaneT.  And yes, Racket also implements CSP as Go, as shown in the table at the beginning of this post.

P.S. As noted in this post, Racket programs build and run much slower than Go programs.

Google Protocol Buffer: Custom Options with Repeated Fields

We encountered a question about Google protocol buffer: could we create a custom option which has a repeated decoration?

The protocol buffer compiler, protoc, answers: no.

However, it is OK to create an option with a message type, and the message could contain a repeated field. Here follows an example program.

The content of learn.proto:

import "google/protobuf/descriptor.proto";

message Something {
  repeated int32 a = 1;
}

extend google.protobuf.FieldOptions {
  optional float my_field_option = 50002;
  optional Something my_another_field_option = 50003;
}

message MyMessage {
  optional string my_field = 1
    [(my_field_option) = 1.1];
  optional string my_another_field = 2
    [(my_another_field_option) = { a : [111, 222, 333] } ];
}

The content of learn.cc:

#include <iostream>
#include <google/protobuf/descriptor.h>
#include "learn.pb.h"

int main() {
  MyMessage m;
  const ::google::protobuf::Descriptor* msg_desc = m.descriptor();
  {
    const ::google::protobuf::FieldDescriptor* field_desc =
      msg_desc->FindFieldByName("my_field");
    const ::google::protobuf::FieldOptions& field_options =
      field_desc->options();
    float option = field_options.GetExtension(my_field_option);
    std::cout << "my_field_option = \n" << option << "\n";
  }
  {
    const ::google::protobuf::FieldDescriptor* field_desc =
      msg_desc->FindFieldByName("my_another_field");
    const ::google::protobuf::FieldOptions& field_options =
      field_desc->options();
    Something option =
      field_options.GetExtension(my_another_field_option);
    std::cout << "my_another_field_option = \n"
              << option.DebugString() << "\n";
  }

  return 0;
}

The content of Makefile:

learn : learn.pb.h learn.pb.cc learn.cc
	g++ -o learn learn.cc learn.pb.cc -lprotobuf && ./learn

learn.pb.h : learn.proto
	protoc learn.proto --cpp_out=. \
          --proto_path=/Users/wyi/Downloads/protobuf-2.4.1/src/:.

The execution result is as so:

my_field_option =
1.1
my_another_field_option =
a: 111
a: 222
a: 333

Few more words here. The field option in protocol buffer has been inherited by the Go programming language and becomes struct tag. As described in the Go language specification:

A field declaration may be followed by an optional string literal tag, which becomes an attribute for all the fields in the corresponding field declaration. The tags are made visible through a reflection interface but are otherwise ignored.

As you see, struct tags in Go are restricted to be strings — no repeated decoration, no message type.

Xcode 4.3 Command Line Tools

After installing new Xcode 4.3, I found that common tools including gcc and g++ are lost.  That is because Xcode does not install these tools into standard locations (/usr/bin nor /usr/local/bin).  But the fix is simple:

1. Install “Command Line Tools” in Xcode : Preferences > Downloads > Command Line Tools > Install
2. Download and install command line tools from https://developer.apple.com/. You need to login but it is free.  And you do not need to install Xcode.

Another looks-good solution is

export PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin:/Applications/Xcode.app/Contents/Developer/usr/bin:$PATH

But it does not work completely. For example, you might suffer from the difficulties including cannot find system header files.

Jailbreaking iPhone: 把 iPhone 变成一台开发工作站

iPhone 和 iPad 内部基本上是 CMU 的 Mach 内核上运行着的 NetBSD 操作系统服务。从这些出色的操作系统技术来看， iPhone 和 iPad 不逊色于任何一款电脑。

自从买了 iPhone 4，我一直在琢磨着怎么怎么把它变得跟一台开发用的电脑一样，比如像装了开发环境（Homebrew）的 Mac MINI 一样。从 Apple Store 上买了一根 iPhone/iPad to VGA 适配电缆之后，我的 iPhone 可以输出到电视和显示器。在 jailbreak 工具（redsn0w）和 Cydia 的帮助下，我可以把 iPhone 和无线键盘、无限鼠标连接起来。我还安装了 iPhone 上的 Mobile Terminal  和一些常用的工具程序，包括和 awk 和 vim。随后，感谢牛鼻的黑客们提供了 GCC 4.2 和 Google Go compiler 的 iPhone 版本。然后呢，用 iPhone 写程序大概就是这个样子的（我其实可以把 iPhone 横过来，这样更能充分利用电视宽屏的空间；图中配角男还应该减减肥）：

在 iPhone 上用 vim 编辑 C 语言和 Go 语言程序的样子如下：

 

编译和运行程序的样子是这样的：

 

附上一些秘笈：

• Cydia 里默认的 Mobile Terminal 版本低，在 iOS 5 上会crash。安装 up-to-date 的 Mobile Terminal：http://iphonejailbreaks.org/mobile-terminal-crashes-on-ios-4-3-1-here-is-the-solution/
• 在手机上用 Mobile Terminal，如果没有无限键盘，敲键太慢。通常是通过电脑 ssh 到 iPhone 的：http://www.cultofmac.com/32134/access-your-iphone-file-system-with-ssh-jailbreak-superguide/
• 在 Apple Store 上购买 iPhone/iPad 的 VGA 转接电缆：http://store.apple.com/us/product/MC552ZM/B
• 在 Cydia 上安装一个叫 DisplayOut 的软件，把 iPhone 屏幕上的内容通过电缆输出到电视或者显示器上。如果没有安装这个软件，只有部分 iPhone 程序能输出到 VGA。
• 在 Cydia 上安装一个叫 BT Keyboard 的程序让 iPhone 可以连接无线键盘
• 在 Cydia 上安装一个叫 BT Mouse 的程序让 iPhone 可以连接无线鼠标
• 在 iPhone 上安装 GCC 4.2：http://cxwangyi.wordpress.com/2012/01/28/how-to-install-gcc-4-2-on-iphone/
• 在 iPhone 上用 GCC build Google Go 语言的编译器：http://cxwangyi.wordpress.com/2012/01/28/learning-go-go-for-iphoneipad/

Jailbreaking iPhone: How to Install GCC 4.2 on iPhone

Following this blog post: http://blog.syshalt.net/index.php/2010/09/12/compile-c-applications-with-gcc-on-ios-4-iphone/, I installed GCC 4.2.1 on my iPhone and gcc works.  However, g++ does not.  Moreover, above post does not explain how comes the header files and library files for gcc on iPhone.

This post complements above post by adding the part about C++.  First of all, I copy-and-pastes above post.

———————————————-

Here is what you need to do in order to be able to compile and run a C application on iphone:

1. You will need to Jailbreak the iPhone first, search on google for more info.
2. Install OpenSSH from Cydia.
3. Connect iPhone to your wireless network and SSH to it.
4. Download this application using: wget http://www.syshalt.net/pub/iphone/gcc-iphone/fake-libgcc_1.0_iphoneos-arm.deb
5. Install libcc using: dpkg –i fake-libgcc_1.0_iphoneos-arm.deb
6. Install iphone-gcc using this command: apt-get install iphone-gcc
(you can download this version from my website: iphone-gcc if does not work on your device the one that is installed by default)
7. Download using: wget http://www.syshalt.net/iphone/gcc-iphone/sdk-2.0-headers.tar.gz
8. Untar with command: tar -xvzf sdk-2.0-headers.tar.gz
9. Enter in the new created folder with: cd include-2.0-sdk-ready-for-iphone
10. Copy all files to include folder with command: cp –r * /usr/include
11. Now type: cd .. in order to return to the previous folder
12. Download using: wget http://www.syshalt.net/iphone/gcc-iphone/gcc_files.tar.gz
13. Untar with command: tar -xvzf gcc_files.tar.gz
14. Enter in the new created folder with command: cd gcc_files
15. Copy all files to /usr/lib using command: cp –r * /usr/lib
16. Install ldid to sign the application (this will prevent iOS to kill the application at startup) using: apt-get install ldid
17. Sign your compilet aplication using: ldid –S <application>
18. Run the application using: ./<application>

——————————————–

From step 9 to step 15, it explains how to install C header files and library files. These files can be copied from the iPhone SDK. On my MacBook, I have Xcode 4.1 and iPhone SDK 4.3, which resides in the directory:

/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS4.3.sdk

I just package the usr subdirectory and throw it to my iPhone:

tar cjf iPhoneOS4.3.sdk-usr.tar.bz2 \
/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS4.3.sdk/usr
scp iPhoneOS4.3.sdk-usr.tar.bz2 mobile@:/var/mobile/Media/

Then, on my iPhone, I unpack the tarball and installed the packing using the following commands:

cd /var/mobile/Media
tar xjf iPhone4.3.sdk-usr.tar.bz2
cd /usr/local
sudo ln -s /var/mobile/Media/iPhoneOS4.3.sdk-usr/include
cd /usr/lib
sudo ln -s /var/mobile/Media/iPhoneOS4.3.sdk-usr/lib/libSystem.B.dylib
sudo ln -s /var/mobile/Media/iPhoneOS4.3.sdk-usr/lib/libSystem.dylib
sudo ln -s /var/mobile/Media/iPhoneOS4.3.sdk-usr/lib/libgcc_s.1.dylib
sudo rmdir system
sudo ln -s /var/mobile/Media/iPhoneOS4.3.sdk-usr/lib/system
for i in  /var/mobile/Media/iPhoneOS4.3.sdk-usr/lib/libstdc++*; do \
sudo ln -s $i; done

It is now the time to test g++. Using the following command:

g++ hello.cc -o hello \
-I /usr/local/include/c++/4.2.1/ \
-I /usr/local/include/c++/4.2.1/armv7-apple-darwin10

I successfully built the following simple C++ program:

#include <stdio.h>

int main() {
        std::cout << "Hello World!";
        return 0;
}

Using Clang: C++0x Features

I am using Ubuntu Linux 10.10, with which, comes Clang based on LLVM 2.8. When I use Clang to build google-test, it complains that “variadic template” is a C++0x feature and is not yet supported.

However, the Clang installed by Homebrew on my Mac OS X 10.7 (Lion) is based on LLVM 3.0 and works well with google-test.

The homepage of LLVM listed pre-built Clang and LLVM 3.0 for Ubuntu 11.04 and 11.10. So I guess I would have to upgrade to at least Ubuntu 11.04.

FreeBSD 9.0 comes with Clang based on LLVM 3.0.

Using Clang: Linkers with Different Platforms

I am using clang++ as the linker with both Ubuntu Linux and Mac OS X. The Ubuntu version invokes collect2, which has parameters –whole-archive and –no-whole-archive to force linking libraries, whereas the Mac OS X version invokes ld (llvm based), which has -force_load for the same purpose.

Using Clang: Cannot Find Standard C++ Library

When I use clang as the linker to link a C++ program, it complains cannot find standard symbols like std::cout. But if I use clang++ as the linker, it works. However, on both Ubuntu and Mac OS X, clang++ is just a symbolic link to clang! I guess when clang runs, it checks the name in which it was started and work accordingly.