-
Notifications
You must be signed in to change notification settings - Fork 1
/
README.txt
154 lines (95 loc) · 4.55 KB
/
README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
Welcome to libfairydust!
libfairydust is a wrapper around NVIDIAs cuda and OpenCL implementation.
AMDs `stream' (APP) OpenCL implementation is also supported
Report bugs and problems to <[email protected]>
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
FAQ Index:
1.0 Why should i use libfairydust?
1.1 Great! How can i compile libfairydust?
2.0 How to use libfairydust
2.1 Playing with the enviroment (local testing)
2.2 Using fairyd (cluster installation)
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
1.0 Why should i use libfairydust?
--------------------------------------------------
libfairydust was written to be used in GPU-Clusters where multiple GPUs
are attached to one physical host and each GPU can be used by a different
user.
Most people do stuff like this:
...
nvret = cudaSetDevice(0)
if(nvret != CUDA_SUCCESS) {
die_in_some_fata_way("oh noes!");
}
...
This works great on your local workstation with one GPU but fails in
spectacular ways on a GPU cluster. In the best case your code will simply crash.
In the worst case *all* your processes will just run on GPU#0 and you won't
even know about it.
Libfairydust solves this problem by simulating `fake-devices': If an application
requests GPU#0, libfairydust can re-route all requests to some other GPU!
This requires *NO* changes to existing applications: The application won't even
know about this: libfairydust works 100% transparent.
1.1 Great! How can i compile libfairydust?
--------------------------------------------------
Libfairydust has been tested with Cuda 3.2 and gcc 4.1.2 + 4.4.4 on x86_64 linux. Support for 32bit installations is currently not implemented.
Compiling it is as easy as doing:
$ cd libfairydust/src
$ make
This should produce a file called ./obj/libfairydust.so.0
(Note: The makefile expects to find all cuda headers at
/usr/local/cuda/include OR/AND at $(CUDA_INSTALL_PATH)/include)
To compile an AMD-Compatible version run:
$ cd libfairydust/src
$ make atidust
This should produce a file called ./obj/libatidust.so.0
(Note: The rest of this documentation will talk about
`libfairydust.so.0' - the ati version works in exactly
the same way - just replace the library name when needed)
2.0 How to use libfairyudst
--------------------------------------------------
libfairydust.so.0 needs to get pre-loaded by ld-linux.
To do this, run something like this in your shell:
$ export LD_PRELOAD=/path/to/obj/libfairydust.so.0
That's it: Libfairydust will now hijack all cuda/openCL calls!
2.1 Playing with the environment (local testing)
--------------------------------------------------
Running any cuda/openCL application with libfairydust preloaded
should now show such an error message:
$ ./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
fairydust.c starting up - (C) 2010-2011 Adrian Ulrich <[email protected]>
fairydust.c release git-108, compiled with gcc-4.4.4 and cuda-3020 at 1295351458
connection to fairyd failed: aborting
Booh! What happened? Libfairyd asked 'fairyd' for some new GPUs but fairyd isn't
running yet.
Don't worry: We can use libfairydust without a running fairyd instance:
$ export FDUST_FORCE_DEBUG=1 # get some debug infos
$ FDUST_ALLOCATE=0 ./deviceQuery
`deviceQuery' will now show exactly ONE GPU (even if you have 10 GPUs)
If you have multiple GPUs you can try to run
$ FDUST_ALLOCATE=1 ./deviceQuery
It will still show only one GPU, but this time libfairydust tricked libcuda into
beliving that GPU#1 is actually GPU#0
You might also have noticed that libfairydust changed the device names into something
like this.
Device = GeForce 9500 GT - fdust{v:h}={0:1}
In this case libfairydust simulated a VIRTUAL (v) device 0.
The REAL (h (hardware)) device for this virtual device is set to device 1
`FDUST_ALLOCATE' also supports multiple GPUs: Setting it to
"0 3 2"
Will map:
The REAL GPU0 to the VIRTUAL GPU0
The REAL GPU3 to the VIRTUAL GPU1
The REAL GPU2 to the VIRTUAL GPU2
Setting FDUST_ALLOCATE="@" will cause libfairydust to return all found GPUs
2.2 Using fairyd (cluster installation)
--------------------------------------------------
In a 'real-world' installation you will have to adapt fairyd.pl to your batchsystem.
The included version of fairyd.pl works with LSF 7.x and was written for use on the
Brutus-Cluster of the ETH Zurich.
Blabla: Fixme: Write more text :-)
- add startup workflow (connect to -> reply)
- protocol description
- testing implementation (use xterm as launcher?)