posted by Rich Wareham on Wed 20th Apr 2005 19:25 UTC
IconI have written (and continue to write) an application called Desktop Manager for OS X. It provides virtual desktops to those refugees from Unix like myself :). Recently it became apparent that it would no longer work under Apple's forthcoming Tiger release of OS X. This proved a little bit of a problem for me since I have no copy of Tiger to test with and don't wish to risk Apple's wrath by obtaining a less-than-legitimate copy. Instead I have relied on a network of people I correspond with via my blog to tell me when things break. It also means I have to attempt to reverse engineer something I don't have in front of me. I am indebted to these people and would like to thank them for their hard work.

I am often asked in email how I uncovered the API calls I use in Desktop Manager which are, unfortunately, undocumented. This article aims to give a little insight into the techniques I use to reverse engineer OS X in order to provide extra functionality to users and extra information to third-party developers. In this article all the utilities I use are a standard part of OS X's developer tools which are freely available. Commercial disassemblers may offer more features but I hope this article will give a feel for how things can be done with free tools.

I shall assume some basic familiarity with the concepts of machine-code programming (i.e. registers, load and store instructions, etc) but in-depth knowledge of PPC assembler is not required.

The problem

One of the features that Desktop Manager has is the ability to use a transition effect when switching virtual desktops. These transitions, according to initial reports I was getting, were one of the things that didn't work in Tiger. This was particularly unfortunate since a non-trivial number of people use Desktop Manager for precisely this feature. The existing code for transitions essentially boils down to this previously reverse engineered code:

void doSwitch(int transition, int direction, float seconds)
  CGSConnectionRef cid = _CGSDefaultConnection();
  CGSSetWorkspaceWithTransition(cid, workspaceNumber, transition, direction, seconds);

Perhaps a little explanation of these lines is in order; each graphical OS X application talks to a 'Window Server', which is responsible for actually compositing the windows on screen, via a 'CoreGraphics connection'. The _CGSDefaultConnection() function returns a reference to the default connection used for the application. The second function uses this connection to talk to the Window Server and switches to workspace (desktop) number 'workspaceNumber' using a particular transition type, direction and duration (in seconds).

This method works beautifully in Panther but doesn't in Tiger. Repeated fiddling with the arguments to CGSSetWorkspaceWithTransition() didn't appear to get any where so it was time to find a new solution. I also wanted this solution to work on both Panther and Tiger. It was time to put my Reverse Engineer T-shirt on and get a little dirty.

Finding a way in

All was not lost as I remembered that during the Panther install there was an introduction movie that plays on the first boot up and then segues into the registration screen via the well-known 'Apple cube' transition. Those I knew with legitimate copies of Tiger reported that the Tiger install has a similar stage and therefore my aim was to find out how this application (the Setup Assistant) did the transition in Panther since I hoped that Apple would be less likely to change the APIs it uses itself when developing Tiger.

I, of course, had no guarantee of this but short of obtaining Tiger illicitly, or waiting for my pre-ordered copy to arrive, it was the only plan of attack I had.

The first stage was to see what other API calls there could be which dealt with transitions. All the undocumented APIs used in Desktop Manager come from the CoreGraphics framework buried inside the System folder. To see what else there was in there the nm tool was used to grep the symbols defined by the framework:

persephone:~ rjw57$ cd /System/Library/Frameworks/ApplicationServices.framework/Frameworks/
persephone:~ rjw57$ nm -g CoreGraphics.framework/CoreGraphics | grep -i transition
965521d8 T _CGSInvokeTransition
96553468 T _CGSNewTransition
96554b1c T _CGSReleaseTransition
965571dc T _CGSSetTransitionAlias
96557d78 T _CGSSetWorkspaceWithTransition
96558bd0 T _CGSTileTransitionPoint

The '-g' flag tells nm to print out only the external symbols (i.e. those that can be linked to trivially). There was some good news here, lots of likely looking API calls. The next step was to see if the Setup Assistant used any of these calls. To do this the otool utility was used to disassemble the Setup Assistant framework:

persephone:~ rjw57$ cd /System/Library/PrivateFrameworks/SetupAssistant.framework/
persephone:~ rjw57$ otool -tV SetupAssistant >~/sadiss.txt

The '-tV' flags tell otool to disassemble and resolve symbols, i.e. make the disassembled code a bit easier to read. A little grep-ing later and a useful-looking Objective C method was found which used some of the interesting API calls.

Developing the solution

Before it can be seen if a particular API call will be useful one needs to work out how to call it. There is no public documentation for these functions and hence no header files that can be used. The first stage in analysis is working out how Setup Assistant calls these functions. Below are a set of extracts from the disassembly which were used to investigate the API. Each block relates to an interesting API call or argument thereof. We shall go through each block in turn and finish with a piece of C code which uses the API calls we are investigating.

Firstly, however, it may be worth going over the PPC calling convention. When calling a function the values for each argument are placed in general purpose registers starting with r3. For example, if we have the following function

int foo(int a, int b, int c)

then upon entry r3 contains the value of 'a', r4 contains the value of 'b' and r5 contains the value of 'c'. The return value is placed in r3 when the function exits. When examining our disassembly we need to pay close attention to what happens to the values of r3 upwards because of this.

The start of the method looks like this (I have added some comments to aid legibility):

; Start of method
-[NSAssistantController(Private) showNewPage:direction:]:
91f4313c	mfcr	r2
91f43140	mfspr	r0,lr
91f43144	bl	saveFP

This is pretty much standard boilerplate code for an Objective C method. Our first interesting call is in this block of code:

; Block 1
91f43218	bl	0x91f670cc	; symbol stub for: __CGSDefaultConnection
91f4321c	lwz	r2,0x478(r1)
91f43220	stw	r3,0x378(r1)

This code calls _CGSDefaultConnection() which from previous work we know takes no parameters and stores r3 (the return value) in a particular memory location (r1 + 0x378). We can re-code this in C fairly simply:

/* Block 1 */
int t1;     /* Temp. variable stored at r1 + 0x378 */
t1 = _CGSDefaultConnection();

This block is interesting because it sets up some memory used later.

; Block 2
91f4323c	li	r4,
91f43240	li	r5,0x14
91f43244	addi	r3,r1,0x40
91f43248	bl	0x91f670ac	; symbol stub for: _memset

We know from existing documentation that memset takes three arguments, a pointer to a region of memory, a value to set all bytes in that memory to and the length of the region. Looking at the disassembly we see that the area of memory to clear is at location r1 + 0x40 and its size is 0x14 (20 bytes or 5 32-bit integers). Our C version is therefore something like:

Table of contents
  1. "Reverse engineering OSX, Page 1/2"
  2. "Reverse engineering OSX, Page 2/2"
e p (0)    38 Comment(s)

Technology White Papers

See More