A Brief Tutorial on Reverse Engineering OS X

I have written (and continue to write) an application called Desktop Manager for OS X. It provides virtual desktops to those refugees from Unix like myself :). Recently it became apparent that it would no longer work under Apple’s forthcoming Tiger release of OS X. This proved a little bit of a problem for me since I have no copy of Tiger to test with and don’t wish to risk Apple’s wrath by obtaining a less-than-legitimate copy. Instead I have relied on a network of people I correspond with via my blog to tell me when things break. It also means I have to attempt to reverse engineer something I don’t have in front of me. I am indebted to these people and would like to thank them for their hard work.

I am often asked in email how I uncovered the API calls I use in Desktop Manager which are, unfortunately, undocumented. This article aims to give a little insight into the techniques I use to reverse engineer OS X in order to provide extra functionality to users and extra information to third-party developers. In this article all the utilities I use are a standard part of OS X’s developer tools which are freely available. Commercial disassemblers may offer more features but I hope this article will give a feel for how things can be done with free tools.

I shall assume some basic familiarity with the concepts of machine-code programming (i.e. registers, load and store instructions, etc) but in-depth knowledge of PPC assembler is not required.

The problem

One of the features that Desktop Manager has is the ability to use a transition effect when switching virtual desktops. These transitions, according to initial reports I was getting, were one of the things that didn’t work in Tiger. This was particularly unfortunate since a non-trivial number of people use Desktop Manager for precisely this feature. The existing code for transitions essentially boils down to this previously reverse engineered code:

void doSwitch(int transition, int direction, float seconds)
  CGSConnectionRef cid = _CGSDefaultConnection();
  CGSSetWorkspaceWithTransition(cid, workspaceNumber, transition, direction, seconds);

Perhaps a little explanation of these lines is in order; each graphical OS X application talks to a ‘Window Server’, which is responsible for actually compositing the windows on screen, via a ‘CoreGraphics connection’. The _CGSDefaultConnection() function returns a reference to the default connection used for the application. The second function uses this connection to talk to the Window Server and switches to workspace (desktop) number ‘workspaceNumber’ using a particular transition type, direction and duration (in seconds).

This method works beautifully in Panther but doesn’t in Tiger. Repeated fiddling with the arguments to CGSSetWorkspaceWithTransition() didn’t appear to get any where so it was time to find a new solution. I also wanted this solution to work on both Panther and Tiger. It was time to put my Reverse Engineer T-shirt on and get a little dirty.

Finding a way in

All was not lost as I remembered that during the Panther install there was an introduction movie that plays on the first boot up and then segues into the registration screen via the well-known ‘Apple cube’ transition. Those I knew with legitimate copies of Tiger reported that the Tiger install has a similar stage and therefore my aim was to find out how this application (the Setup Assistant) did the transition in Panther since I hoped that Apple would be less likely to change the APIs it uses itself when developing Tiger.

I, of course, had no guarantee of this but short of obtaining Tiger illicitly, or waiting for my pre-ordered copy to arrive, it was the only plan of attack I had.

The first stage was to see what other API calls there could be which dealt with transitions. All the undocumented APIs used in Desktop Manager come from the CoreGraphics framework buried inside the System folder. To see what else there was in there the nm tool was used to grep the symbols defined by the framework:

persephone:~ rjw57$ cd /System/Library/Frameworks/ApplicationServices.framework/Frameworks/
persephone:~ rjw57$ nm -g CoreGraphics.framework/CoreGraphics | grep -i transition
965521d8 T _CGSInvokeTransition
96553468 T _CGSNewTransition
96554b1c T _CGSReleaseTransition
965571dc T _CGSSetTransitionAlias
96557d78 T _CGSSetWorkspaceWithTransition
96558bd0 T _CGSTileTransitionPoint

The ‘-g’ flag tells nm to print out only the external symbols (i.e. those that can be linked to trivially). There was some good news here, lots of likely looking API calls. The next step was to see if the Setup Assistant used any of these calls. To do this the otool utility was used to disassemble the Setup Assistant framework:

persephone:~ rjw57$ cd /System/Library/PrivateFrameworks/SetupAssistant.framework/
persephone:~ rjw57$ otool -tV SetupAssistant >~/sadiss.txt

The ‘-tV’ flags tell otool to disassemble and resolve symbols, i.e. make the disassembled code a bit easier to read. A little grep-ing later and a useful-looking Objective C method was found which used some of the interesting API calls.

Developing the solution

Before it can be seen if a particular API call will be useful one needs to work out how to call it. There is no public documentation for these functions and hence no header files that can be used. The first stage in analysis is working out how Setup Assistant calls these functions. Below are a set of extracts from the disassembly which were used to investigate the API. Each block relates to an interesting API call or argument thereof. We shall go through each block in turn and finish with a piece of C code which uses the API calls we are investigating.

Firstly, however, it may be worth going over the PPC calling convention. When calling a function the values for each argument are placed in general purpose registers starting with r3. For example, if we have the following function

int foo(int a, int b, int c)

then upon entry r3 contains the value of ‘a’, r4 contains the value of ‘b’ and r5 contains the value of ‘c’. The return value is placed in r3 when the function exits. When examining our disassembly we need to pay close attention to what happens to the values of r3 upwards because of this.

The start of the method looks like this (I have added some comments to aid legibility):

; Start of method
-[NSAssistantController(Private) showNewPage:direction:]:
91f4313c	mfcr	r2
91f43140	mfspr	r0,lr
91f43144	bl	saveFP

This is pretty much standard boilerplate code for an Objective C method. Our first interesting call is in this block of code:

; Block 1
91f43218	bl	0x91f670cc	; symbol stub for: __CGSDefaultConnection
91f4321c	lwz	r2,0x478(r1)
91f43220	stw	r3,0x378(r1)

This code calls _CGSDefaultConnection() which from previous work we know takes no parameters and stores r3 (the return value) in a particular memory location (r1 + 0x378). We can re-code this in C fairly simply:

/* Block 1 */
int t1;     /* Temp. variable stored at r1 + 0x378 */
t1 = _CGSDefaultConnection();

This block is interesting because it sets up some memory used later.

; Block 2
91f4323c	li	r4,__register_frame_info.eh
91f43240	li	r5,0x14
91f43244	addi	r3,r1,0x40
91f43248	bl	0x91f670ac	; symbol stub for: _memset

We know from existing documentation that memset takes three arguments, a pointer to a region of memory, a value to set all bytes in that memory to and the length of the region. Looking at the disassembly we see that the area of memory to clear is at location r1 + 0x40 and its size is 0x14 (20 bytes or 5 32-bit integers). Our C version is therefore something like:

/* Block 2 */
int pt2[5]; /* An array which starts at r1 + 0x40 */
memset(pt2, 0, 20); /* 20 = 0x14 */

where I’ve avoided checking what value the memory gets cleared with since it isn’t interesting at this moment in time.

Our next block is interesting because it calls one of the API functions we don’t know about.

; Block 3
91f43314	addi	r4,r1,0x40
91f43318	lwz	r3,0x378(r1)
91f4331c	addi	r0,r2,0x74
91f43320	addi	r5,r1,0x370
91f43324	stw	r0,0x50(r1)
91f43328	bl	0x91f6708c	; symbol stub for: _CGSNewTransition

Straight away we can see that only r3, r4 and r5 are set implying that this API call takes 3 arguments. The first argument, r3, is loaded from the memory location r1 + 0x378. Referring back to block 1 we see that this is the value returned from _CGSDefaultConnection(). The second argument, r4, is set to point to memory location r1 + 0x40. As we have seen above, this is a 20 byte area of memory previously cleared by memset and then modified by the code between blocks 2 and 3. The final argument, r5, is set to point to memory location r1 + 0x370. Since this isn’t really set anywhere in the disassembly we can assume that this is a pointer to some temporary variable. We can now write a C version of this block:

/* Block 3 */
int t3;     /* Temp. variable stored at r1 + 0x370 */
CGSNewTransition(t1, pt2, &t3);

We now investigate the last two blocks which contain interesting API calls.

; Block 4
91f433f0	lwz	r4,0x370(r1)
91f433f4	cmpwi	cr7,r4,__register_frame_info.eh
91f433f8	beq+	cr7,0x91f43404
91f433fc	lwz	r3,0x378(r1)
91f43400	bl	0x91f66fec	; symbol stub for: _CGSReleaseTransition
; Block 5
91f43484	lwz	r3,0x378(r1)
91f43488	addis	r2,r31,0x3
91f4348c	lwz	r4,0x370(r1)
91f43490	lfs	f1,0xde60(r2)
91f43494	bl	0x91f66fcc	; symbol stub for: _CGSInvokeTransition
91f4357c	mtcrf	56,r11
91f43580	b	restFP
; End of method

By examining the values of r3 and r4 and comparing with block 3 we can easily write equivalent C for these blocks:

/* Block 4 */
CGSReleaseTransition(t1, t3);

/* Block 5*/
CGSInvokeTransition(t1, t3, f1);

where f1 is some floating point argument. Comparing it with CGSSetWorkspaceWithTransition() we can probably be fairly sure it is the duration of the transition in seconds. The hard work is now over. We’ve discovered the number of arguments each API call takes and where their values come from. The remaining task is to combine these arguments with what we already know to give them names more meaningful than t1, etc.

From the call to _CGSDefaultConnection() it is clear that t1 holds a reference to the application’s connection to the Window Server. The variable t3 is set by CGSNewTransition() and used by CGSReleaseTransition() and CGSInvokeTransition(). We can therefore suppose that it is some handle which refers to the created transition. The memory at pt2 is passed only to CGSNewTransition() so we can suppose that it is a structure which specifies the various parameters of the transition. The call CGSReleaseTransition(), interestingly, happens before CGSInvokeTransition(). It seems unlikely that both of these functions would be called in that order so we may suppose that the transition is only released if some other operation fails and that invoking the transition implicitly releases it. Using this information we can flesh our our C version:

CGSConnectionRef cid;
int transitionHandle;
int transitionSpec[5];
float duration;

cid = _CGSDefaultConnection();
memset(transitionSpec, 0, 20); /* 20 = 0x14 */
/* ... modifiy contents of transitionSpec (between blocks 2 and 3) ... */
CGSNewTransition(cid, transitionSpec, &transitionHandle);
/* ... find out if we failed ... */
       CGSReleaseTransition(cid, transitionHandle);
       /* set duration */
       CGSInvokeTransition(cid, transitionHandle, duration);

All that remains now is to work out the format of transitionSpec. After a little trial and error we find the following code works just as well as our initial code:

CGSConnectionRef cid;
int transitionHandle;
int transitionSpec[5];

cid = _CGSDefaultConnection();

/* Set up the transition specification */
memset(transitionSpec, 0, 20); /* 20 = 0x14 */
transitionSpec[1] = transition;
transitionSpec[2] = direction;

/* Set up the transition itself */
CGSNewTransition(cid, transitionSpec, &transitionHandle);
/* Switch desktop */
CGSSetWorkspace(cid, workspaceNumber);
/* A little pause to let the Window Server sort itself out ... */
/* ... and fire off the transition */
CGSInvokeTransition(cid, transitionHandle, seconds);


The above code was added to Desktop Manager and a version sent to those people with Tiger. To my great relief (and suprise) people reported that transitions were working! There are still problems to be worked out but, because of this bit of reverse engineering, Desktop Manager on Tiger will soon be a reality.

Can we now write some documentation on these new API calls? Indeed we can. A little experimentation has shown that these are not just useful for virtual desktops. Indeed anyone can use them in a full-screen app to provide Keynote-style transitions. Simply call CGSNewTransition() which will freeze the screen, draw your new screen and call CGSInvokeTransition() to reveal it to the user.

About the author:
Rich Wareham is a 3rd year PhD student in the Signal Processing Group of the Cambridge University Engineering Department. When not playing with Geometric Algebra and proving theorems he enjoys learning about technology, experimenting with new Operating Systems and teaching undergraduates about Software Engineering.

If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.


  1. 2005-04-20 7:38 pm
  2. 2005-04-20 7:41 pm
  3. 2005-04-20 7:44 pm
  4. 2005-04-20 7:46 pm
  5. 2005-04-20 7:56 pm
  6. 2005-04-20 8:06 pm
  7. 2005-04-20 8:08 pm
  8. 2005-04-20 8:11 pm
  9. 2005-04-20 8:12 pm
  10. 2005-04-20 8:15 pm
  11. 2005-04-20 8:19 pm
  12. 2005-04-20 9:43 pm
  13. 2005-04-20 9:47 pm
  14. 2005-04-20 9:50 pm
  15. 2005-04-20 10:10 pm
  16. 2005-04-21 12:01 am
  17. 2005-04-21 12:15 am
  18. 2005-04-21 12:15 am
  19. 2005-04-21 1:36 am
  20. 2005-04-21 1:41 am
  21. 2005-04-21 1:47 am
  22. 2005-04-21 2:27 am
  23. 2005-04-21 2:33 am
  24. 2005-04-21 2:47 am
  25. 2005-04-21 3:33 am
  26. 2005-04-21 4:10 am
  27. 2005-04-21 4:13 am
  28. 2005-04-21 7:09 am
  29. 2005-04-21 7:48 am
  30. 2005-04-21 11:23 am
  31. 2005-04-21 3:43 pm
  32. 2005-04-21 3:46 pm
  33. 2005-04-21 4:13 pm
  34. 2005-04-21 4:58 pm
  35. 2005-04-21 5:59 pm
  36. 2005-04-21 7:18 pm
  37. 2005-04-26 12:00 am
  38. 2005-04-28 12:45 pm