Q-Max的雜談: 2016

2016年9月28日星期三

TensorFlow Census predict

如果得到如下的結果

Traceback (most recent call last):

File "census.py", line 22, in <module>

print (df_train["income_bracket"].apply(lambda x: ">50K" in x))

File "/usr/lib/python2.7/dist-packages/pandas/core/series.py", line 2023, in apply

mapped = lib.map_infer(values, f, convert=convert_dtype)

File "inference.pyx", line 920, in pandas.lib.map_infer (pandas/lib.c:44780)

File "census.py", line 22, in <lambda>

print (df_train["income_bracket"].apply(lambda x: ">50K" in x))

TypeError: argument of type 'float' is not iterable

在read_csv時用skipfooter=2 跳過最後兩行

2016年7月20日星期三

Install Nvidia driver with dkms may help you

DKMS is Dynamic Kernel Module Support, which can help you when you upgrade your kernel with the out come modules.

To install Nvidia Graphic Card driver with DKMS support, install DKMS first.

$ sudo apt-get install dkms

Then install the driver, take NVIDIA-Linux-x86_64-367.35.run for example

$ chmod +x NVIDIA-Linux-x86_64-367.35.run
$ sudo sh NVIDIA-Linux-x86_64-367.35.run --dkms

When the installation finish

$ dkms status

You can find "nvidia, 367.35, 3.19.0-64-generic, x86_64: installed".

And you can build all modules for current kernel by

$ dkms autoinstall

And for a specifi kernel by

$ dkms autoinstall -k 4.4.15

Or install a module for all kernels

$ dkms install -m nvidia -v 367.35 --all

Reference https://wiki.archlinux.org/index.php/Dynamic_Kernel_Module_Support

Nvidia Fan speed control

Login by Xwindow
Run nvidia-xconfig cool-bits=4
Then either run the bash file below or open Nvidia X Server Setting to change the fan speed

The bash file

#!/bin/bash
nvidia-settings -a [gpu:0]/GPUFanControlState=1
nvidia-settings -a [fan:0]/GPUTargetFanSpeed=100

2016年7月18日星期一

compile kernel with minimum modules

Use the option localmodconfig to minize the modules.
But any change in hardware will make it need to be compiled again.

# get linux kernel source code
$ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.4.15.tar.xz
$ tar -xvf linux-4.4.15.tar.xz
$ cd linux-4.4.15
$ make mrproper
$ cp /boot/config-$(uname -r) .config
$ make oldconfig
$ make localmodconfig
# compile kernel. revision is the tag you can customize. -j4 means using 4 threads
$ sudo fakeroot make-kpkg --initrd --revision=4.4.localmodconfig kernel_image kernel_headers -j4
# wait for a while

# you will find linux-headers-4.4.15_4.4.localmodconfig_amd64.deb and linux-image-4.4.15_4.4.localmodconfig_amd64.deb outside
$ ls ../linux*
# install them
$sudo dpkg -i ../linux*localmodconfig_amd64.deb

2016年7月13日星期三

Tensorflow's docker image with gpu supported doesn't work

Nvidia driver要先裝好

https://github.com/NVIDIA/nvidia-docker/wiki/Installation

先裝docker-engine

https://docs.docker.com/engine/installation/

再裝nvidia-docker

https://github.com/NVIDIA/nvidia-docker

使用Tensorflow Docker image(注1)

$ nvidia-docker pull gcr.io/tensorflow/tensorflow:latest-gpu

$ nvidia-docker run -it gcr.io/tensorflow/tensorflow:latest-gpu sh -c 'python -m tensorflow.models.image.mnist.convolutional'

結果 not find libcudnn.so

離開 container (Ctrl+D)

進入 container，使用bash

$ nvidia-docker run -it gcr.io/tensorflow/tensorflow:latest-gpu bash

連結

$ ln -s /usr/lib/x86_64-linux-gnu/libcudnn.so.4 /usr/lib/x86_64-linux-gnu/libcudnn.so

Run again

$ python -m tensorflow.models.image.mnist.convolutional

結果

https://www.tensorflow.org/versions/r0.9/get_started/os_setup.html#docker-installation

注1： If cannot connect to the Docker daemon. Is the docker daemon running on this host?

http://stackoverflow.com/questions/33562109/docker-command-cant-connect-to-docker-daemon

2016年7月7日星期四

global variable in bash

ether write in /etc/profile or add a sh in /etc/profile.d/

so making a file /etc/profile.d/cuda.sh

export LD_LIBRARY_PATH=/usr/local/cudnn/lib64:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}

export PATH=/usr/local/cuda/bin:${PATH}

export CUDA_HOME=/usr/local/cuda

would make the variables for all users

reference:http://serverfault.com/questions/491585/is-there-a-global-bash-profile-for-all-users-on-a-system

2016年6月12日星期日

其實有很多教學了，有個小小的點是，如果你有多顯卡，在關閉開源社群的driver nouveau的時候可能會完全沒有畫面，但其實已經開好機在等你login了

Through there are many web pages teach you how to install cuda on ubuntu, I encounter a problem that I have multiple vga cards(HD4000 and 660ti) in my computer, when I follow their instruction to close the driver, nouveau, from open source community, then I boot with whole black screen. Actually, the computer is there waiting for you to login!

So I use SSH to continue the installation.

我個人的作法
Download cuda_7.5.18_linux.run from Nvidia website

$ chmod +x cuda_7.5.18_linux.run
$ sudo ./cuda_7.5.18_linux.run

Follow the installation
It tell you that you have to reboot because nouveau is working
It actually make a file in /etc/modprobe.d/nvidia-installer-disable-nouveau.conf
And its content is

blacklist nouveau
options nouveau modeset=0

And it run "$ sudo update-initramfs -u" to modify the modules in initramfs.

OK, you know what it have done to your computer now, and let's reboot.

If you find that there is whole black waiting for you, and you don't have ssh installed, try recovery mode, I just give keywords here.
boot in root shell

$ mount [sdXy] / -o remount,rw # [sdXy] is your disk partition where / locate
$ rm /etc/modprobe.d/nvidia-installer-disable-nouveau.conf
$ update-initramfs -u
$ reboot

Then you can boot normally into the familiar ubuntu desktop, and then install ssh., then run the previous procedure again.

OK, we have ssh now. Use other computer to login the poor computer.

$ sudo service lightdm stop # This doesn't do anything for me.
$ sudo killall xorg # But this does.
$ sudo ./cuda_7.5.18_linux.run

Follow the instructions to install Nvidia driver and CUDA.

Add the two line to the bottom of the file .bashrc in your home directory.

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH}
export PATH=/usr/local/cuda/bin:${PATH}

Then

$ source ~/.bashrc

Remember to run following command to have a good xconfig file for your card, or it may still become a workstation that you are only able to access through ssh or telnet.

$ nvidia-xconfig

Then reboot, bless you to see your familiar ubuntu desktop again.

2016年6月8日星期三

ipython and cs231n

Ubuntu 16.04 server

sudo pip install ipython[notebook]
sudo pip install scipy

sudo apt-get install libjpeg-dev zlib1g-dev # These are dependency for Pillow
sudo pip install Pillow # This is a dependency for scipy.misc imread
sudo apt-get install libpng12-dev libfreetype6-dev libxft-dev # for matplotlib
sudo pip install matplotlib

2016年6月1日星期三

Memory hierarchy and cache

Principles

smaller is faster. (in cache view point)
a program property is locality.

Temporal locality : referenced data tend to be referenced again soon. (e.g. loops)

Spatial locality : if some data are referenced, data beside the referenced data tend to be referenced soon.

Hit Time << Miss Penalty so can Memory Hierarchy exist.

Registers - L1 Cache - L2 Cache - L3 Cache - DRAM - Disk

Registers are managed in code generation, which is by compiler.
Caches are managed by hardware.
DRAM and Disk are managed by OS.

Why Virtual Memory is fully associative?

Disk is too slow. Calculation and then placing a page in a good place doesn't take too much time. The penalty to replace them between Disk and DRAM is much higher than calculation.

Block placement : fully associative, direct mapped(1-way associative), set-associative
Write strategy : Write through, Write back

Write through will write to lower layer simultaneously when writing in upper layer, e.g. when write to cache, it write to main memory simultaneously. It need write buffer to queue the writes to memory.

Write back means you can write many times in upper layer and it become dirty state, but it don't write to memory until this block become victim block and need to be replace.

Write around, ............

手動設定Ubuntu的nameserver(dns)、固定IP

To set remote dns server manually, we have to modify the file, "/etc/network/interface" .

To set 8.8.8.8 as our name server, add "dns-nameservers 8.8.8.8" to the file.

And to set the static IP, modify the file "/etc/network/interface" as well.