the problem
So I work at a lab, where CUDA classes are taught. We purchased a few GPUs for the classroom a few months ago : 4 Nvidia GTX970 and a GTX980. Last monday, a teacher told me about his CUDA code not running anymore. I had a similar issue on a server and it was caused by a driver incompatible with a kernel update, so I tried updating the driver. It didn't help. I spent a day and a half trying to fix the issue, until i had a hunch and decided to actually look inside the machines. Guess what. Some asshole stole the GPUs inside 3 computers and replaced hem with older ones.
The thing is, this classroom has restricted access: only a few students have keys, and access to the classroom is automatically logged (electronic keys with an access control server somewhere). So if I can pinpoint the moment the GPUs where changed, security can compare the dates with the rooms access logs.
I'm still a novice and I'm not sure how to do that exactly. Would you help me catch the thief ? Because of this guy, there are no more CUDA classes for anyone !
what I have tried
I have used lspci to get the GPU's PCI address, and then grepped for that address in the kernel log. I got a bunch of lines starting like this:
Apr 11 14:30:34 HOSTNAME kernel: [ 0.305476] pci 0000:04:00.0: [10b5:8613] type 01 class 0x060400 I noticed that the second string behind brackets ( [10b5:8613] in this example) is always the same up to a point, then changes and stays the same. I'm guesing it's some sort of device ID, but I coulndn't find more info about it.
On one of the 3 computers, this entry is followed by another one that gives the start and finish addresses of the VRAM area assigned to the device. I wrote a quick python script to compute the size of that area, and noticed that it get significantly smaller on the same day the string in brackets changes.
Additional info
- 3 out of 5 GPUS were stolen
- a GTX970 and the GTX980 have been replaced by GTX295
- a GTX970 has been replaced by a GTX9800
- I have copies of the logs for the three machines with me
- The machines are all running Ubuntu 14.04 Desktop
- The theft occured max. a month ago
EDIT: Thanks to advice from /u/mercenary_sysadmin, we have pinpointed the date of the GPUs swap - a sunday evening in the middle of the holidays. I will go to the Police on monday, wich will allow the university's security team to cross-check the room's access logs. I will edit with an update then.
[link][49 comments]