Jex’s Note

Ruby Basic

變數

以此例為例子

class Var
  def print
    puts $hh
  end
  $hh = "hh"
end

t = Var.new
t.print

替換以下變數

  • $name : 全域變數. 結果: 正常
  • @name : 實例變數, 作用僅限於 self 指示的物件. 結果: 相當於輸出 nil, 什麼東沒有
  • @@name : 類別變數, 在 class 內使用, 如果另個物件繼承它的物件, 也可以使用 @@name. 結果: 正常
  • name : 區域變數 (小寫字母或底線開頭, 初始化前並沒有 nil 值). 結果: undefined local variable or method hh'
  • Name : 常數 (大寫開頭, 不可重覆定義). 結果: 正常

結論 : 定義在 class 內的變數必須是 全域, 類別或常數

宣告

陣列

%w(i1 b2 c3 j4)                 # ["i1", "b2", "c3", "j4"]

arr = {                         # {:A => 1, :B => 2 }
    'A' : 1,
    'B' : 2,
}

基本 class 觀念

class Foo

  def instance_method                           # instance method   (可以用 self 取 instance 的值)
    'instance_method 要先 new 才能使用'
  end

  def self.class_method                         # class method (無法用 self 取內部的值)
    'slef.class_method 不能 new, 可直接使用'
  end

  def call_method
    inside_method
  end

  protected

  def inside_method
    'Call 內部的 inside_method 不需加 `self.` prefix'
  end

end

a = Foo.new
puts a.instance_method      # instance_method 要先 new 才能使用
puts Foo.class_method       # slef.class_method 不能 new, 可直接使用
puts a.call_method          # Call 內部的 inside_method 不需加 `self.` prefix

判斷

0 及 empty 都是 TRUE, 只有 false 與 nil 才是 FALSE

是否為數值

9.9.integer?

數值是否在範圍裡面

(11..15).include? 15
15.between?(11, 15)

檔案相關

Check directory in existence, Create folder

require 'fileutils'
if File.directory?('/tmp/layer1')
  puts "Folder exist !"
else
  puts "Create folder ..."
  FileUtils.mkpath('/tmp/layer1/layer2')
end

Check if a file exists

File.exist?('/tmp/ruby_test/send/sorry.mp3')

Getting path from full file path

2.0.0-p247 :001 > File.dirname("/tmp/ruby_test/send/qq.php")
 => "/tmp/ruby_test/send"

Getting filename from path

2.0.0-p247 :003 > File.basename("/tmp/ruby_test/send/qq.php")
 => "qq.php"

Integer & Float

小數第一位四捨五入

a = 633.633
a.round(1)     # 633.6

次方

2 ** 3

絕對值

-5.abs      # 5

取最近的整數/四捨五入

5.6.round   # 6

取整數/無條件捨去

9.9.floor   # 9

取整數/無條件進位

2312.22.ceil    # 2313

下一個數

2.next      # 3

二元運算

n & num
n | num
n ^ num (XOR)
n << num (向左位移)
n >> num (向右位移)

Rand

rand(1..10)
[true, false].sample

亂數

SecureRandom.random_number(99999999999)             # 9997979524

建立 UUID

require 'securerandom'                  # 原生
SecureRandom.hex(20)                    # ac5d23f916dd83dcc495dc5f0b8b602942b635fa    長度會是你傳值的 2 倍
SecureRandom.base64(20)                 # WcLJKJzgibphbiXHKIeCsP8jtN8=

SecureRandom : This library is an interface for secure random number generator which is suitable for generating session key in HTTP cookies

字串處理

gsub

"hello".gsub(/[aeiou]/, '*')                    # "h*ll*"
"hello".gsub(/([aeiou])/, '<\1>')               # "h<e>ll<o>"
"hello".gsub(/./) {|s| s.ord.to_s + ' '}        # "104 101 108 108 111 "
"hello".gsub(/(?<foo>[aeiou])/, '{\k<foo>}')    # "h{e}ll{o}"
'hello'.gsub(/[eo]/, 'e' => 3, 'o' => '*')      # "h3ll*"

split

'C3-0803986E423F0C66DA56'.split('-')            # ["C3", "0803986E423F0C66DA56"]

last word

'example'.last                                  # e

取固定位置及長度的字串

a = "1234567890"
a[1..3]  # 234

Hash

key 是否存在

genders.has_key?(:male)

value 是否存在

genders.value?(2)

刪除某個 key

a.delete('es')

刪除某個 value

hash.delete_if{|_,v| v == "2"}

Reverse key <-> value

Hash[h.to_a.collect(&:reverse)]

Find value and return key

h.key(value)    # return key

Get all keys from Hash

h.keys

Sort

hash = {f: 4, a: 2, r: 1 }
hash.sort # => [[:a, 2], [:f, 4], [:r, 1]]          # 這不是我們要的結果
hash.sort.to_h                                      # => {:a=>2, :f=>4, :r=>1} , 這才是

兩個 hash 合併另種寫法

a = {"zh"=>1, "es"=>5}
b = {"zh"=>1, "qq"=>5}
a.merge(b)                  # {"zh"=>1, "es"=>5, "qq"=>5}

# 這個範例可以看的出它只比對 key,如果 key 一樣值不一樣,則會被後者的值覆蓋
a = {"zh"=>1, "es"=>5}
b = {"zh"=>1, "es"=>7}
a.merge(b)                  # {"zh"=>1, "es"=>7}

只取得 hash 裡面某些 key -> value

{a: 1, b: 2, c: 3, d: 4 }.slice(:a, :b)
{:a=>1, :b=>2}

Array

是否為 array

array.is_a?(Array)

去除相同的值 :

[2,4,2,5,1].uniq            # => [2, 4, 5, 1]

Array to Hash

a = ['apple', 'banana', 'mongo']
c = Hash[a.map { |e| [e.to_sym, e + ' !'] }]
 => {:apple=>"apple !", :banana=>"banana !", :mongo=>"mongo !"}

a = [["Bob", 1], ["Jex", 2], ["Jxx", 3]]
Hash[a.map { |e| [e[0].to_sym, e[1]]}
 => {:Bob=>1, :Jex=>2, :Jxx=>3}

刪除 each 判斷

a.delete_if { |x| x >= 3 }

a.delete_if do |v|
  if v >= 3
    true                # Make sure the if statement returns true, so it gets marked for deletion
  end
end

array’s value to symbol

array.map { |x| x.to_sym }
或
array.map &:to_sym

值是否存在

a = [1,2,3]
a.include?(2)

多項是否存在, 只要其中一項存在就是 true
([2, 6, 13, 99, 27] & [5,6]).any?

找出值旳 index

array.find_index(90)

兩個 array 合併

a.zip(s).flatten.compact
或
s.inject(a, :<<)

add

會改變原值
a.push("d", "e", "f")
a << 'x'

不會改變原值
a + [3]

delete

會改變原值
a = [2,4,6,3,8]
a.delete(3)

不會改變原值
a - [3]

remove duplicate elements

array = array.uniq

join array (無值就 delete value)

my_array.delete_if(&:empty?).join(',')

ordre ASC

my_array.sort

order DESC

my_array.sort.reverse

order 某個欄位

my_array.sort_by &:lastname

associations 關聯,如果想刪除其中某一筆

items.each do |i|
  if i.product.nil?
    needed_remove << i  # 先放進待刪除區
  end
end
items - needed_remove

# 以下看起來會 work ,但實際上最後的 items 還是跟原始的一樣,也就是 delete 沒有在原本的 items 移除該項目
items.each do |i|
  if i.product.nil?
    items.delete(i)
  end
end
items

insert 一筆到最前面

[1,3,6].unshift(5)              # [5, 1, 3, 6]

json

string to json

JSON.parse(params[:JSONData])

each

一行

cats.each(&:name)
cat_names = cats.map(&:name)            # 相當於 each cat.name
cats.each do |cat| cat.name end
cats.each {|cat| cat.name }

Call Dynamic method name

Sometimes you might need a dynamic method, you can use send

send

def test
  puts 'test~'
end

send('test')

前面也可以接 instance 以及可以多個 method 一起執行

User.send('smart').send('honest')           # 等於 User.smart.honest

傳遞參數

User.send('smart', iq: 130)

只 call public method

User.public_send(:name, "Jex")

method call

程式碼比較多而且不能串多個 method

> a = "my_string"
> meth = a.method("size")
> meth.call() # call the size method
=> 9

型態

Print variable type

3.class
 => Fixnum
3.class.superclass
 => Integer

加密

Digest::SHA256.digest 'message'         # "\xABS\n\x13\xE4Y\x14\x98+y\xF9\xB7\xE3\xFB\xA9\x94\xCF\xD1\xF3\xFB\"\xF7\x1C\xEA\x1A\xFB\xF0+F\fm\x1D"

Digest::SHA256.hexdigest('message')     # ab530a13e45914982b79f9b7e3fba994cfd1f3fb22f71cea1afbf02b460c6d1d

Progress bar

簡易

10.times{|i| STDOUT.write "\r#{i}"; sleep 1}

1~100%

progress = 'Progress [ '
1000.times do |i|
  j = i + 1
  if j % 10 == 0
    # 將 = 加到 progress string
    progress << "="

    # 將游標移到這行的最前面
    print "\r"

    # 取代目前這一行, 並輸出進度
    print progress + " #{j / 10} % ]"

    # flush buffer 立即顯示
    $stdout.flush
    sleep 0.05
  end
end
puts "\nDone!"

定義 hash parameters

def instance_method(hash = {})
Call : a.instance_method aa: 'aa', bb: 'bb'

hash key 使用 integer 時注意 :

hash = {1: 'one'} # will not work
hash = {1 => 'one'} # will work

Date format

Symbol

  • %Y: 2017
    • %C : 20 (year / 100)
    • %y : 17 (year % 100)
  • %B : January
    • %^b : JANUARY
  • %b : Jan 也等於 %h
    • %^b : JAN
  • %m : 01..12 with zero-padding
    • %_m : 1..12 with blank-padding
    • %-m : 1..12 without balnk-padding
  • %d : 01..31 with zero-padding
    • %e : 1..31 with blank-padding
    • %-d : 1..31 without balnk-padding
  • %j : 001..336 day of the year
  • %H : 00..23 with zero-padding
  • %k : 0..23 with blank-padding
  • %I : 01..12 with zero-padding
  • %l : 1..12 with blank-padding
  • %P : am / pm
  • %p : AM / PM
  • %M : 00..59 minute
  • %S : 00..59 second
  • %L : 000..999 millisecond
  • %N = 9 digits nanosecond
    • %3N = 3 digits millisecond
    • %6N = 6 digits microsecond
    • %9N = 9 digits nanosecond
    • %12N = 12 digits picosecond
  • %z : +0900
    • %:z : +09:00
    • %::z : +09:00:00
    • %:::z : +09:30:30
    • %Z : CST time zone abbreviation name
  • %A : Sunday
    • %^A : SUNDAY
    • %a : Sun
    • %^a : SUN
    • %u : 1..7 day of the week (monday is 1)
    • %w : 0..6 day of the week (sunday is 0)
  • %G : 2016 the week-based year
    • %g : 16 the last 2 digits of the week-based year
    • %V : 52 Week number of the week-based year
  • %U : 00..53 week number of the year. starts with sunday
    • %W : 00..53 starts with mondy
  • %s : 1483249934 timestamp, number of seconds since 1970-01-01 00:00:00 UTC
  • %n : \n
    • %t : \t
    • %% : %
  • %c : Sun Jan 1 13:54:29 2017 date and time (%a %b %e %T %Y)
    • %D = %x : 01/01/17 Date (%m/%d/%y)
    • %F : 2017-01-01 The ISO 8601 date format (%Y-%m-%d)
    • %v : 1-JAN-2017 VMS date (%e-%b-%Y)
    • %r : 01:56:03 PM 12-hour time (%I:%M:%S %p)
    • %R : 13:56 24-hour time (%H:%M)
    • %T = %X : 13:55:44 24-hour time (%H:%M:%S)

語法

Time.now                                        # 2016-04-07 11:04:09 +0800
Time.now.getutc                                 # 2016-04-07 03:05:00 UTC
Time.now.strftime('%F %T')                      # 2015-08-10 21:02:04
Time.now.strftime('%F %R')                      # 2015-09-21 01:04
Time.now.strftime("%Y-%m-%d %H:%M:%S")          # 2015-08-10 21:01:06
Time.now.in_time_zone                           # Tue, 22 Sep 2015 09:27:56 CST +08:00    如果 config 有設定 timezone
Time.now.to_i                                   # 1483249426 (timestamp)

String 轉回 Time

Time.parse("2015-09-21 12:00")
或
"2015-09-21 12:00".to_time

2 天前 (rails)

2.days.ago

2 天後 (rails)

2.days.since

現在 DateTime 減 5 天

Post.find(12).update(created_at: Time.now.strftime('%F %R').to_time - 5.days)

5 天後

Date.today + 5
Date.today + 5.days

10.days.from_now

date 相減

require 'date'
> now = Date.today
> before = Date.today + 2.days
> difference_in_days = (before - now).to_i

-----
start_date = Date.parse "2012-03-02 14:46:21 +0100"
end_date =  Date.parse "2012-04-02 14:46:21 +0200"
(end_date - start_date).to_i

Add time to date

d = Date.new(2012, 8, 29)
t = Time.now
dt = DateTime.new(d.year, d.month, d.day, t.hour, t.min, t.sec, t.zone)
或
Date.new(2015, 2, 10).to_datetime + Time.parse("16:30").seconds_since_midnight.seconds

Today

Date.today

Year

Time.now.year

顯示時間語意

# console 要引入 include ActionView::Helpers::DateHelper
time_ago_in_words(3.minutes.from_now)                       # => 3 minutes
time_ago_in_words(Time.now - 15.hours)                      # => 15 hours
time_ago_in_words(Time.now)                                 # => less than a minute
from_time = Time.now - 3.days - 14.minutes - 25.seconds     # => 3 days
distance_of_time_in_words('2015-09-16'.to_date, Date.today) # => 不到 1 分鐘

Gem error

Gem::FilePermissionError

$ gem install sitemap_generator
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /usr/local/rvm/gems/ruby-2.2.1 directory.

使用 sudo 安裝也不行

sudo: gem: command not found

解決方法 : 切換到 sudo 加上 gem 相關的 PATH

$ $PATH
-bash: /usr/local/rvm/gems/ruby-2.2.1/bin:/usr/local/rvm/gems/ruby-2.2.1@global/bin:/usr/local/rvm/rubies/ruby-2.2.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

切換到 sudo, 在 .bashrc 最後一行加上

export PATH=$PATH:/usr/local/rvm/gems/ruby-2.2.1/bin:/usr/local/rvm/gems/ruby-2.2.1@global/bin:/usr/local/rvm/rubies/ruby-2.2.1/bin

再重新登入, 完成!

Gem::LoadError: You have already activated rake ...

Gem::LoadError: You have already activated rake 12.0.0, but your Gemfile requires rake 11.3.0.

1) Check your Gemfile to see if rake exists. If yes, upgrade it.

2) If no, execute bundle update rake

防火牆 Iptable

基本設定

/etc/iptables.firewall.rules:

*filter

#  Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT

#  Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allow all outbound traffic - you can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

#  Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allow SSH connections
#
#  The -dport number should be the same port number you set in sshd_config
#
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

#  Allow ping
-A INPUT -p icmp -j ACCEPT

#  Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

#  Drop all other inbound - default deny unless explicitly allowed policy
-A INPUT -j DROP
-A FORWARD -j DROP

COMMIT

附加設定

-A INPUT -p tcp --dport 445 -j ACCEPT    // for samba
-A INPUT -p tcp -s 10.160.55.77     --dport 11211 -j ACCEPT // for memcache
-A INPUT -p tcp -m state --state NEW --dport 4123 -j ACCEPT   // ssh port 改成 4123
-A INPUT -p icmp -j ACCEPT // allow ping

啟動防火牆

sudo iptables-restore < /etc/iptables.firewall.rules

檢查是否設定成功

sudo iptables -L

重開後防火牆自動啟動

編輯 sudo vim /etc/network/if-pre-up.d/firewall:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables.firewall.rules

permission :

sudo chmod +x /etc/network/if-pre-up.d/firewall

Q & A

發生 iptables-restore: line 36 failed

雖然 36 行是指向 COMMIT, 但並不是它的問題, 是前面一些設定的問題,

把下面這些刪除再跑一次看看, 在 Raspberry Pi 發現不刪會一直過不了

#  Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

Golang 建立自己的套件

先確定 GOPATH 路徑存在, 這是放你的 package 檔案位置

.bash_profile :

export GOROOT=/usr/local/go
export GOPATH=$HOME/projects/go/mygo
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

$GOPATH 底下要有這三個目錄, 你應該建立它們

  • src : 存放源始碼 (ex: .go .c .h .s)
  • pkg : 編譯後生成的文件 (ex: .a)
  • bin : 編譯後生成的可執行檔

在 src 下建立一個練習用的套件 exlib

appletekiMacBook-Air-2:mygo apple$ tree src
src
└── exlib
    ├── exlib.go
    └── sub_exlib
        └── sub_exlib.go

2 directories, 2 files

exlib/exlib.go :

package exlib

type TestInterface interface {
    run() string
}

type TestStruct struct {
    Name string
    Age  int
}

// return initial struct
func NewStruct(name string, age int) (testStruct *TestStruct) {
    testStruct = &TestStruct {
        Name: name,
        Age: age,
    }

    return testStruct
}

// Implement TestStruct func
func (testStruct TestStruct) run() string {
    return "run run run!!"
}

// Execute interface's func
func InterfaceFunc (testStruct TestStruct) string {
    testInterface := TestInterface(testStruct)
    return testInterface.run()
}

建立 exlib 的相關子 package

在 exlib 建立子目錄 sub_exlib

exlib/sub_exlib/sub_exlib.go:

package sub_exlib

// Public func
func SubFunc() string {
    return "sub_exlib"
}

你也可以將 sub_exlib 這個 package 分成許多檔案 :

exlib/sub_exlib/sub_exlib2.go:

package sub_exlib

// Public func
func QQ() string {
    return "sub_exlib2"
}

注意必須都要在 exlib/sub_exlib 下以及 package 名稱也都要一樣

完成! 引入並執行剛剛建立的套件

主程式 :

package main

import (
    "fmt"
    "exlib"
    "exlib/sub_exlib"
)

func main() {
    // New struct
    new_struct := exlib.NewStruct("Jex", 26)
    fmt.Println(*new_struct)

    // Interface - implement func, and run its func
    interfaceRun := exlib.InterfaceFunc(*new_struct)
    fmt.Println(interfaceRun)

    // Sub package
    sub_exlib := sub_exlib.SubFunc()
    fmt.Println(sub_exlib)
}

結果 :

{Jex 26}
run run run!!
sub_exlib

ref: https://github.com/jex-lin/build-web-application-with-golang/blob/master/ebook/01.2.md

Golang Basic

宣告

Multiple variables

ff, xx := 3, "cc"

[string]{ .. }

elements := map[string]map[string]string{
    "A": map[string]string{
        "field1": "val1",
        "field2": "val2",
    },
    "B": map[string]string{
        "field1": "val1",
        "field2": "val2",
    },
}

[string]int

map := map[string]int{"apple": 5, "lettuce": 7}

[string]interface{}

x := map[string]interface{}{
    "foo": []string{"a","b"},
    "bar": "foo",
    "baz": 10.4,
}

[]interface{}

hnapCommandMap := make(map[string]interface{})
hnapCommandMap["ID"] = "06"
hnapCommandMap["Info"] = map[string]string{
    "Name":       "Jex"
}

resque2 := map[string]interface{}{
    "class": "hnap",
    "args":  []interface{}{hnapCommandMap},
}

map[string]interface{}

方法1
a := map[string]interface{}{
    "int":    1,
    "string": "xx",
}

方法2
t := map[string]interface{}{}
t["id"] = 312
t["type"] = "realtime"
t["data"] = []map[string]string{
    {
        "did":    "did1",
        "action": "action1",
    },
    {
        "did":    "did2",
        "action": "action2",
    },
}

但不可以, 無法這樣給值
var t map[string]interface{}
t["xxx"] = "xxx"

宣告 slice struct ([]struct)

type Target struct {
    Topic   string `json:"topic"`
    Message string `json:"message"`
}
type Payload struct {
    Targets []Target `json:"targets"`
    From    string   `json:"from"`
}

var payload = Payload{
    Targets: []Target{
        Target{
            Topic:   "topic1",
            Message: "message1",
        },
        Target{
            Topic:   "topic2",
            Message: "message2",
        },
    },
    From: "api",
}

宣告空 slice, var VS :=

宣告出來為 nil, 長度 0 (建議)

var q []string

宣告出來為 [], 長度 0, 如果要 json encode 想避免 null 的話, 建議使用這個

q := []string{}

其他

var t *T = new(T)   // or  t := new(T)
var a uint64 = 22   // 等於 a := uint64(22)
a := 0x12           // 等於 18

const 對應 int (有點像 enum)

const (
    LevelCritical = iota    // 0
    LevelError
    LevelWarning
    LevelNotice
    LevelInfo               // 4
    LevelDebug
)

指標

  • & : 對變數取址
  • * : 對指針取值, 將指標取回一般變數

Example :

package main
import "fmt"
func main() {
    var a int = 1
    var b *int = &a
    var c **int = &b
    var x int = *b
    fmt.Println("a = ",a)                           // 1
    fmt.Println("&a = ",&a)                         // &a =  0xf840037100
    fmt.Println("*&a = ",*&a)                       // *&a =  1
    fmt.Println("b = ",b)                           // b =  0xf840037100
    fmt.Println("&b = ",&b)                         // &b =  0xf840037108
    fmt.Println("*&b = ",*&b)                       // *&b =  0xf840037100
    fmt.Println("*b = ",*b)                         // *b =  1
    fmt.Println("c = ",c)                           // c =  0xf840037108
    fmt.Println("*c = ",*c)                         // *c =  0xf840037100
    fmt.Println("&c = ",&c)                         // &c =  0xf840037110
    fmt.Println("*&c = ",*&c)                       // *&c =  0xf840037108
    fmt.Println("**c = ",**c)                       // **c =  1
    fmt.Println("***&*&*&*&c = ",***&*&*&*&*&c)     // ***&*&*&*&c =  1
    fmt.Println("x = ",x)                           // x =  1
}

將指標轉成實體, 讓 code 更乾淨

func ArgsToJob(Args *map[string]interface{}) Job {
    args := (map[string]interface{})(*Args)
    job := Job{
        ID:          args["ID"].(string),
    }
    return job
}

將 interface{} 轉成 map[string]interface{}

a := (map[string]interface{})(args[0].(interface{}).(map[string]interface{}))

什麼時候用指標?

參考 code :

type User struct {
  Name string
}

func main() {
  u := &User{Name: "Leto"}
  println(u.Name)
  Modify(u)
  println(u.Name)
}

func Modify(u *User) {
  u = &User{Name: "Paul"}
}

如果是比較大的結構,在宣告時可以用指標以節省記憶體, 但高階語言慢慢不用指標了,所以其實也可以不用特別宣告用指標, 用結構 call function 用結構傳遞,傳遞參數給 function 的話則不需要特別用指標,

指標行為

code :

func main() {
    a := QQ{}
    b := &a
    c := *b

    fmt.Printf("a: %p\n", &a)
    fmt.Printf("b: %p (address same as a) \n", b)
    fmt.Printf("c: %p (different address from a, b)\n", &c)

    d := xx()
    fmt.Printf("d: %p (different address from x)\n", &d)

    e := zz()
    fmt.Printf("e: %p (different address from z)\n", &e)
}

func xx() QQ {
    x := QQ{}
    fmt.Printf("x: %p (return instance)\n", &x)
    return x
}

func zz() *QQ {
    z := QQ{}
    fmt.Printf("z: %p (return pointer)\n", &z)
    return &z
}

如果 struct 定義沒有欄位, 結果 :

type QQ struct {}

a: 0x1127a88
b: 0x1127a88 (address same as a)
c: 0x1127a88 (different address from a, b)
x: 0x1127a88 (return instance)
d: 0x1127a88 (different address from x)
z: 0x1127a88 (return pointer)
e: 0xc42000c030 (different address from z)

如果 struct 定義有欄位, 結果 :

type QQ struct {
    Name string
}

a: 0xc42000e290
b: 0xc42000e290 (address same as a)
c: 0xc42000e2a0 (different address from a, b)
x: 0xc42000e2e0 (return instance)
d: 0xc42000e2d0 (different address from x)
z: 0xc42000e2f0 (return pointer)
e: 0xc42000c030 (different address from z)

關於回傳 instance or pointer 的記憶體 : call func 拿到的都是新的記憶體, 不管 func 裡面回傳的是不是指標

swtich

map + switch

m := map[string]int{"foo":1}
f := func(key string) bool { _, ok := m[key]; return ok }
switch {
    case f(key):
        // whatever

or

switch category {
case
    "auto",
    "news",
    "sport",
    "music":
    return true
}

斷言 (type assertion)

分辨型別

var anything interface{} = "string"
switch v := anything.(type) {
case string:
    fmt.Println(v)
case int32, int64:
    fmt.Println(v)
case interface{}:
    fmt.Println(v)
default:
    fmt.Println("unknown")
}

已知道是什麼型別

value, ok := a.(string)
if !ok {
    fmt.Println("It's not ok for type string")
    return
}

// or
if str, ok := a.(string); ok {

轉型

int to int64

i64 := int64(23)

int to float64

i := 5;
f := float64(i)

int to string

s := strconv.Itoa(123)

int64 to int

i := int64(5)
int(i)

int64 to string

s := strconv.FormatInt(int64(5), 10)

int64 to float64

float64(1)

int64 to uint64

u, err := strconv.ParseUint(strconv.FormatInt(int64(123), 10), 10, 64)

float64 to string

s64 := strconv.FormatFloat(v, 'E', -1, 64)

float64 to int64

var f float64 = 55.3
i = int64(f)

float64 to uint

uint(user["age"].(float64))

string to byte

dd := "dcf"
fmt.Println([]byte(dd))

result : [100 99 102]

string to int

v, err = strconv.Atoi(s)

string to int64

v, err := strconv.ParseInt(s, 10, 64)

string to uint32/uint64

v := "42"
if s, err := strconv.ParseUint(v, 10, 32); err == nil {
    fmt.Printf("%T, %v\n", s, s)
}
if s, err := strconv.ParseUint(v, 10, 64); err == nil {
    fmt.Printf("%T, %v\n", s, s)
}

string to float64

v, err := strconv.ParseFloat("55.74", 64)

*string to string

value := *pointer_str

byte to string

s := string(byteArray)
s := string(byteArray[:])

rune to string

r := rune('a')
fmt.Println(reflect.TypeOf(r))  // int32
fmt.Println(r, string(r))       // 97 a

array to slice

x[:]

指標(pointer)轉實體

*f is msg := sqs.Message(*m)

(*f)["cc"]
or
(map[string]interface{})(*f)["cc"]
or
msg := sqs.Message(*m)

interface{} to int

a := job["retryTimes"].(int)

[]interface{} 轉成 interface{}

interface{}([]interface{}{reactor, sensor})

[]interface{} to []int ref : InterfaceSlice

var dataSlice []int = foo()
var interfaceSlice []interface{} = make([]interface{}, len(dataSlice))
for i, d := range dataSlice {
    interfaceSlice[i] = d  // 或用 append 的方式
}

interface{} conver to bytes.Buffer

switch v := any.(type) {
case bytes.Buffer:          // return as is
    return v.String()       // Here v is of type bytes.Buffer
}

interfacer{} (Server) to (Server)

type Server struct {
    Name string
}

var v interface{}
if v == nil {
    v = &Server{Name: "xx"}
}
fmt.Println(reflect.TypeOf(v))      // *main.Server
s := v.(*Server)
fmt.Println(s.Name)                 // xx

map[string]interface{} 變成陣列 []interface{}

[]interface{}{reactor, sensor}

struct to another struct

type A struct{ Name string }
type B struct{ Name string }
a := A{"aa"}
b := B(a)
b.Name = "bb"
fmt.Println(a)
fmt.Println(b)

bytes to io.Reader

bytes.NewReader(b)

bytes.Buffer to io.Writer

var b bytes.Buffer
writer := bufio.NewWriter(&b)

strings to io.Reader

strings.NewReader(s)

file to bytes

b, err := ioutil.ReadFile("/tmp/ff.tmp")

file to io.Writer

file, err = os.Open("/tmp/ff.tmp")
defer file.Close()

image to bytes

buf := new(bytes.Buffer)
err := jpeg.Encode(buf, new_image, nil)
send_s3 := buf.Bytes()

convert to another interface

dimg, ok := img.(draw.Image)

loop

Infinite

for {

}

Range, like other languages

for i:=1; i<=5; i++ {

}

Until the specific time

for time.Now().Unix() < 1481089195 {
    time.Sleep(1 * time.Second)
}

slice

slice結構

結構

     []byte
     ---------
ptr  | *elem |
     ---------
len  |  int  |
     ---------
cap  |  int  |
     ---------

a slice with five elements

s := make([]byte, 5)  // 指定 len, 不指定 cap

     []byte
     ---------
ptr  |       |  -> [5] bytes | 0 | 0 | 0 | 0 | 0 |
     ---------
len  |  5    |
     ---------
cap  |  5    |
     ---------

nil slice

var s []byte

     []byte
     ---------
ptr  |  nil  |
     ---------
len  |   0   |
     ---------
cap  |   0   |
     ---------

cap and len

  • cap: capacity of the underlying array.
  • len: how many items are in the array.

The slice abstraction in Go will resize the underlying array for you.

Example:

s := make([]int, 0, 3)
for i := 0; i < 5; i++ {
    s = append(s, i)
    fmt.Printf("cap %v, len %v, %p\n", cap(s), len(s), s)
}

output:

cap 3, len 1, 0xc000088000
cap 3, len 2, 0xc000088000
cap 3, len 3, 0xc000088000
cap 6, len 4, 0xc00006a030
cap 6, len 5, 0xc00006a030

基本操作

append item

list = append(list, item)

append slice

list = append(list, list2...)

其他

重新切一個 slice, 新 slice 會使用原本 slice 的底層, 改成使用 copy 才會是一個新的 slice

func main() {
    data := get()
    fmt.Println(len(data), cap(data), &data[0])
}

func get() []byte {
    raw := make([]byte, 10000)
    fmt.Println(len(raw), cap(raw), &raw[0])
    return raw[:3]
}
// 10000 10000 0xc42005e000
// 3 10000 0xc42005e000     (記憶體位置一樣)

func get() (res []byte) {
    raw := make([]byte, 10000)
    fmt.Println(len(raw), cap(raw), &raw[0])
    res = make([]byte, 3)
    copy(res, raw[:3])
    return
}
// 10000 10000 0xc42005e000
// 3 3 0xc42000e280

傳遞 slice 是傳遞記憶體位置

感謝 chris:

  • 記得 slice map 的變數 只是一個參照
  • slice 跟 map 的變數 參照的 值都是隔開的
  • 但是底層運作是使用 指標
  • 裡面真正的值 是存在記憶體
  • 關鍵字: 淺拷貝/深拷貝

example

func main() {
    d := []string{"xxx"}
    fmt.Println("original:", d)
    nonPointer(d)
    fmt.Println("non-pointer:", d)
    change(d)
    fmt.Println("change:", d)
    pointer(&d)
    fmt.Println("pointer:", d)
}

func nonPointer(d []string) {
    d = append(d, "vvvv")
}

func change(d []string) {
    d[0] = "ccc"                // 可修改到外部
}

func pointer(d *[]string) {
    *d = append(*d, "vvvv")
}

結果

original: [xxx]
non-pointer: [xxx]
change: [ccc]
pointer: [ccc vvvv]

宣告獨立乾淨的 slice

不論是參數傳遞 (call by value) 還是 assign 給一個新的變數, *elem (pointer) 都會指向同一個, 而改其中一個也會改動到另一個

如果沒有 append 新的 item, 只改變原本 slice 長度內的值 e.g. s[2] = 3 (如果s總長度是5), 都會一直是共用指標

如果要打破這個規則, 就將其中一個 append 新的 item 那麼 *elem (pointer) 就會是不同的

如果要將 slice 指給一個新的變數, 但又不想要將原本的 *elem (pointer) 參照到原本的變數, 除了用 make 也可以 new 全新的再用 append 給值

code

s := []int{1, 2, 3}

// Old *elem
s1 := s

// New *elem
s2 := s
s2 = append(s2, 4)

// New *elem
s3 := []int{}
s3 = append(s3, s...)

// New *elem
s4 := make([]int, 0)
s4 = append(s4, s...)

s[1] = 999

fmt.Println(s)      // [1 999 3]
fmt.Println(s1)     // [1 999 3]
fmt.Println(s2)     // [1 2 3 4]
fmt.Println(s3)     // [1 2 3]
fmt.Println(s4)     // [1 2 3]

array vs slice

差別主要是 array 是有固定長度, slice 沒有

code:

// array
var List = [2]int{1, 2}  // or [...]int{1,2}
copyList := List
copyList[1] = 4
fmt.Printf("new=%v, old=%v\n", copyList, List)

// slice
var oldSlice = []int{1, 2, 3, 4}
newSlice := oldSlice
newSlice[3] = 10
newSlice = append(newSlice, 100)
fmt.Printf("new=%v, old=%v\n", newSlice, oldSlice)

result:

new=[1 4], old=[1 2]
new=[1 2 3 10 100], old=[1 2 3 10]

array 是直接操作記憶體位置

check if two slices are the same

reflect.DeepEqual(tmp, item.ans)

map misc

傳入接收 actionData map[string]interface{} 參數的 func

send_var(jobData["ActionData"].(map[string]interface{}))

func send_var(actionData []interface{}) (err error) {
    fmt.Println(actionData[0].(map[string]interface{})["Did"])

[]map[string]interface{} 傳入型態接受 []interface{} 的 func

test(&map[string]interface{})
...
func test(rec *[]interface{})

用指標方式傳入可避免型態轉換發生的問題

map[string]interface{} 一直輸出它的 index 順序不一定會一樣

qq := map[string]interface{}{
    "0": "111111",
    "1": "xxxxxxxx",
}

[]interface{} 一直輸出它的 index 順序會一樣

用 mi 定義 type, 讓 code 更乾淨

type mi map[string]interface{}
res["Envelope"].(interface{}).(mi)["Body"].(interface{}).(mi)[actionData["Name"].(string)+"Response"].(interface{})

assignment to entry in nil map : 會發生此原因是你在賦值給 map 時沒有初始化

var d []map[string]interface{}

(Wrong)
    var f map[string][]map[string]interface{}
    f["ss"] = d

(Correct)
    f := map[string][]map[string]interface{}{}
    f["ss"] = d

(Correct)
    var f map[string][]map[string]interface{}
    f = make(map[string][]map[string]interface{})
    f["ss"] = d

map 是否為空

if len(map) == 0 {
    ....
}

判斷 key 是否存在, 如果不存在不會造成 error

a := map[string]interface{}{
    "a": "A",
}

if t1, matched := a["a"]; matched {
    fmt.Println(t1.(string))
}

if t2 := a["a"]; t2 != nil {
    fmt.Println(t2.(string))
}

判斷 key 及型態

a := map[string]interface{}{"fff": "xx"}
ff, ok := a["fff"].(int)   // 即使來源跟對象的型態不一樣, 不會造成 panic
if !ok {
    fmt.Println("no")
    // return
}
fmt.Println(ff)

// result
no
0

Slice Tricks

判斷 slice key 是否存在

a := []string{"zero", "one", "two"}
fmt.Println(a, len(a))
x, v := 3, "nothing"
if len(a)-1 >= x  {
    v = a[x]
}
fmt.Printf("%s", v)

type *map[string]interface {} does not support indexing

func tt(dd *map[string]interface{}) {
    (*dd)["qq"] = "qqqq"
}

傳遞 map 是傳址非傳值

func main() {
    dd := map[string]interface{}{
        "ff": "ffff",
    }
    fmt.Printf("original, mem: %p val: %v\n", &dd, dd)
    nonPointer(dd)
    fmt.Printf("non-pointer, val: %v\n", dd)
    change(dd)
    fmt.Printf("change, val: %v\n", dd)
    pointer(&dd)
    fmt.Printf("pointer, val: %v\n", dd)
    newMap(dd)
    fmt.Printf("newMap, val: %v\n", dd)
    makeMap(dd)
    fmt.Printf("makeMap, val: %v\n", dd)
}

// 會修改到外部
func nonPointer(dd map[string]interface{}) {
    fmt.Printf("non-pointer, mem: %p\n", &dd)
    dd["dd"] = "ddd"
}

// 會修改到外部
func change(dd map[string]interface{}) {
    fmt.Printf("non-pointer, mem: %p\n", &dd)
    dd["ff"] = "cccc"
}

// 會修改到外部
func pointer(dd *map[string]interface{}) {
    fmt.Printf("pointer, mem: %p\n", &dd)
    (*dd)["qq"] = "qqqq"
}

// 即使 qq 裡, new 一個新變數 ff, 並將 dd 的值給 ff, 修改仍會改到外面傳進來的 map
func newMap(dd map[string]interface{}) {
    ff := dd
    ff["ff"] = "zzzz"
}

// 必須要用 make new 一個實體, 才不會參照到原本的記憶體位址
func makeMap(dd map[string]interface{}) {
    cc := make(map[string]interface{})
    for k, v := range dd {
        cc[k] = v
    }
    cc["bbb"] = "bbb"
}

結果

original, mem: 0xc42000c028 val: map[ff:ffff]
non-pointer, mem: 0xc42000c038
non-pointer, val: map[ff:ffff dd:ddd]
non-pointer, mem: 0xc42000c040
change, val: map[ff:cccc dd:ddd]
pointer, mem: 0xc42000c048
pointer, val: map[ff:cccc dd:ddd qq:qqqq]
newMap, val: map[dd:ddd qq:qqqq ff:zzzz]
makeMap, val: map[dd:ddd qq:qqqq ff:zzzz]

func params

傳入/接收 map pointer

error: (type *map[string]interface {} does not support indexing)

solution:

func main() {
    a := map[string]interface{}{
        "xxx": "XXX",
    }

    qq(&a)
}

func qq(tmp *map[string]interface{}) {
    fmt.Println((*tmp)["xxx"])
}

傳入/接收 map pointer

var args = map[string]interface{}{
    "A": "1",
    "B": "2",
}
foo(&args)
func foo(args *map[string]interface{}) {
    options := (map[string]interface{})(*args)
    fmt.Println(options["A"])
}

foo(args)
func foo(name string, ...args map[string]interface{}) {
    options := (map[string]interface{})(args[0])
    fmt.Println(options["A"])
}

傳入 Optional 參數

T1("A", "B")
func T1(str ...string) {
    fmt.Println(str[0]) // A
}

T2([]string{"A", "B"})
func T2(str ...[]string) {
    fmt.Println(str[0]) // [A B]
}

map

test("test")
test("test", map[string]interface{}{"dd": "DD", "cc": "CC"})

func test(required string, optional ...map[string]interface{}) {
    fmt.Println(required)
    fmt.Println(len(optional))
    if len(optional) > 0 {
        if val, existed := optional[0]["dd"]; existed {
            fmt.Println(val)
        }
    }
}

將 slice params 傳入 args…

func echo(strings ...string) {
    for _, s := range strings {
        fmt.Println(s)
    }
}

func main() {
    strings := []string{"a", "b", "c"}
    echo(strings...) // Treat input to function as variadic
}

interface & struct

[struct] 成員大小寫

跟 func 一樣,大寫代表 public,小寫是 private

[struct] map

type AWS struct {
   SQS  map[string]*aws_sqs.SQS
}

var qq AWS
qq.SQS = make(map[string]*aws_sqs.SQS)
qq.SQS[xx] = &ws_sqs.SQS{ ... }

[struct] function pointer

type Job struct {
    Done func() error
}

var job Job
job.Done = done

func done() error {

}

[struct] 定義 + 賦值

resp := Music{
    Genre: struct {
        Country string
        Rock    string
    }{
        Country: "Taylor Swift",
        Rock:    "Aimee",
    },
}

Assign a struct to an interface

type Interface interface{}

type Struct struct{}

func main() {
        var ps *Struct
        var pi *Interface
        pi = new(Interface)
        *pi = ps

        _, _ = pi, ps
}

關於 interface 及 struct 的用法

package main

import "fmt"

# 用 interface 定義一個抽象層,只負責說這個類別有什麼動作
type PeopleAction interface {
    AnimalAction
    Stand()
}

# 本身也可以被繼承
type AnimalAction interface {
    Run()
    Eat()
}

# 定義資料結構
type Animal struct {
    Name string
}

func (animal *Animal) Run() {
    fmt.Println(animal.Name + " is running")
}

func (animal *Animal) Eat() {
    fmt.Println(animal.Name + " is eatting")
}

func (animal *Animal) Stand() {
    fmt.Println(animal.Name + " is standing")
}

func main() {
    var jex PeopleAction = &Animal{"Jex"}
    var bob AnimalAction = &Animal{"Bob"}
    jex.Run()               // Jex is running
    jex.Eat()               // Jex is eatting
    jex.Stand()             // Jex is standing
    bob.Run()               // Bob is running
    bob.Eat()               // Bob is eatting
    // bob.Stand()          // bob.Stand()  Error : bob.Stand undefined (type AnimalAction has no field or method Stand)
}

Receive interface as param

type A interface{}

type B struct {
    A
    Age int
}

func main() {
    var a A
    fmt.Println("interface A type: ", reflect.TypeOf(a))
    b := B{a, 30}
    fmt.Println("struct B type: ", reflect.TypeOf(b))
    Show(b)
}

func Show(a A) {
    fmt.Println("show age: ", a.(B).Age)
    fmt.Println("interface param type: ", reflect.TypeOf(a.(B)))
}

Result

interface A type:  <nil>
struct B type:  main.B
show age:  30
interface param type:  main.B

Type T satisfies interface I defined in the first snippet. Values of type T can be f.ex. passed to any function accepting I as a parameter

type I interface {
    M() string
}
type T struct {
    name string
}
func (t T) M() string {
    return t.name
}
func Hello(i I) {
    fmt.Printf("Hi, my name is %s\n", i.M())
}
func main() {
    Hello(T{name: "Michał"}) // "Hi, my name is Michał"
}

single type can implement many interfaces

type I1 interface {
    M1()
}
type I2 interface {
    M2()
}
type T struct{}
func (T) M1() { fmt.Println("T.M1") }
func (T) M2() { fmt.Println("T.M2") }
func f1(i I1) { i.M1() }
func f2(i I2) { i.M2() }
func main() {
    t := T{}
    f1(t) // "T.M1"
    f2(t) // "T.M2"
}

different structs, call same name but different func

type I interface {
    M() string
}
type T1 struct{ Name string }
type T2 struct{ Name string }

func main() {
    // Different structs call its own func
    var i I

    t1 := T1{Name: "t1"}
    t2 := T2{Name: "t2"}
    i = &t1 // Must use pointer
    fmt.Println(i.M())
    i = &t2
    fmt.Println(i.M())
}

func (t *T1) M() string { return t.Name + " T1" }
func (t *T2) M() string { return t.Name + " T2" }

same interface, different structs, call same func

No, you can’t!

Difference between passing by reference and passing by value

var a = &A{Val: "aaa"}
fmt.Println("a: ", a)
fmt.Println("b = a")
b := a
b.Val = "bbb"
fmt.Println("b: ", b, " (set new value to b)")
fmt.Println("a: ", a, " (affected by b)")

var c = &A{Val: "ccc"}
fmt.Println("c: ", c)
fmt.Println("b = c")
b = c
b.Val = "bbbbbbbbbbb"
fmt.Println("b: ", b, " (set new value to b)")
fmt.Println("c: ", c, " (affected by b)")
fmt.Println("a: ", a, " (unaffected)")

Result:

a:  &{aaa}
b = a
b:  &{bbb}  (set new value to b)
a:  &{bbb}  (affected by b)
c:  &{ccc}
b = c
b:  &{bbbbbbbbbbb}  (set new value to b)
c:  &{bbbbbbbbbbb}  (affected by b)
a:  &{bbb}  (unaffected)

struct + json

json.Marshal(struct {
        Data     *MyStruct   `json:"data,omitempty"`
        Status   string      `json:"status,omitempty"`
        Reason   string      `json:"reason,omitempty"`
    }{
        Data:   &r.Data,
        Status: r.Status,
        Reason: r.Reason,
    })
}

Assign func

type Job struct {
    ID string
    Do func(*Job)
}

func main() {
    var j1 = &Job{ID: "J01", Do: do}
    var j2 = &Job{ID: "J02", Do: do}
    j1.Do(j1)
    j2.Do(j2)
}

func do(j *Job) {
    fmt.Println("Job ID:", j.ID)
}

巢狀

type Resp struct {
    UserID struct {
    } `json:"user_id"`
    Data struct {
        Info struct {
            Name        string   `json:"name"`
            Friends     []Friend `json:"friends"`
        } `json:"info"`
    } `json:"data"`
}

輸出

"user_id": {},
"data": {
    "info": {
        "name": "XXX",
        "friends": [
            { },
            { },
        ],
    },
},

傳遞參數不像 map 及 slice 以指標傳遞, 而是傳遞實體

func main() {
    q := qq{name: "ori"}
    fmt.Println(q)
    dd(q)
    fmt.Println(q)
    cc(&q)
    fmt.Println(q)
}

func dd(q qq) {
    q.name = "xx"
}

func cc(q *qq) {
    q.name = "zz"
}

result

{ori}
{ori}
{zz}

JSON tag 補充

// 接收跟輸出都會忽略
Field int `json:"-"`

// Field appears in JSON as key "myName" and the field is omitted from the object if its value is empty, as defined above.
Field int `json:"myName,omitempty"`

// Field appears in JSON as key "Field" (the default), but the field is skipped if empty.
Field int `json:",omitempty"`

type Ticket struct {
    Places []Place `json:"places"`  // json 傳過來時的 places 也要是陣列
    Name string `json:"name"`       // 單純接收字串
    UID string                      // 注意! 即使沒設定 tag, 如果傳進來是 "uid": "123" 這樣也會 map 得到
    FriendID string                 // 如果傳進來是 "friend_id": "123" 這樣 map 不到
}

Nested struct 即使有設 omitempty 也 json 仍會輸出 "field": {}, 解決方法是指定 pointerstruct

type A struct { Val  L2 `json:"val,omitempty"` }
type B struct { Val *L2 `json:"val,omitempty"` }        // pointer 是關鍵
type L2 struct { Val string `json:"val,omitempty"` }

json.Marshal 輸出 :

a := &A{Val: L2{Val: ""}}   // {"val":{}}
a := &A{}                   // {"val":{}}
a := &B{}                   // {}

避免空的 slice json marshal 後為 null

var users []map[string]interface{}
if users == nil {
    users = make([]map[string]interface{}, 0)
}

import

運作流程

當 import A 時, 會先執行 A 的 init(), 但如果 A 裡面還有 import B, 則會優先執行 B 的 init(),

當 package 的 init() 都執行完後, 才會執行 main 的 init(), 最後才會執行 main()

import 的幾種類型

1) 正常引入

import(
    "fmt"
)

這是我們平常最常引入的方法

2) 省略前綴的 package name

import (
    . "fmt"
)

ex : fmt.Println("hello world") 可以省略寫成 Println("hello world")

3) Alias

import (
    f "fmt"
)

ex : f.Println("hello world")

4) 不引用 package 內部函數

import (
    _ "fmt"
)

當引入這個 package 只會執行它的 init()

套件管理

dep (官方)

  • dep init: 第一次才會用到, 會產生 Gopkg.toml, Gopkg.lock (不要理它), vendor (安裝好 package)
  • dep ensure: install, Gopkg.toml 裡面設定的 package. 另外如果要新增套件到 Gopkg.toml 也是用這個指令, 不用指定套件它會自已去掃
  • dep ensure -update: 升級所有套件到最新的
  • dep ensure -add github/aaa/bbb@{tag or commit_id}: 新增套件到 Gopkg.toml / Gopkg.lock and vendor
  • dep status: 顯示你目前的版本跟最新的版本
  • 升級某個套件版本要改 Gopkg.toml, 再執行 dep ensure

Vendor

govendor,用起來挺直覺的,如果使用過 npm 或 gem 的經驗,它的概念是一樣的

govendor cheat sheet

第一次使用

govendor init               # 會產生 vendor/vendor.json, 但裡面是空的
govendor add +external      # 幫你把目前使用到 GOPATH 套件版本填到 vendor.json,也會產生 github.com, etc. 第三方套件

第 N 次使用, 更新 govendor.json

govendor remove +v          # 清空 `vendor.json` 及刪除 `github.com`, .. etc.
govendor add +external      # 再重加有使用到的套件

依據 vendor/vendor.json 使用到的套件從 local 的 GOPATH 加到 vendor/, 如果 local 沒有要先下 go get

govendor sync

如果 github 某個套件更新了, 想更新 vendor 這個套件

govendor fetch golang.org/x/net/context
Error: Remotes failed for

如果有個 local repo (private repo) 加不進 vendor/

$ govendor sync
Error: Remotes failed for:
        Failed for "relog" (failed to ping remote repo): unrecognized import path "relog"

直接 add 那個 local repo

govendor add relog
升級某一套件版本
govendor fetch github.com/satori/go.uuid

(不要採用, 僅保留)

  1. 先從 govendor 移除掉 govendor remove github.com/parnurzeal/gorequest
  2. 移除 local 的 github.com/parnurzeal/gorequest
  3. go get github.com/parnurzeal/gorequest
  4. govendor add +external 更新到 vendor.json
不管怎麼重 build, code 一直是舊的

有可能你將目前專案加到 vendor/ 下了, 把它刪掉, 再重編就可以了

Panic and Recover

攔截並印出 error

如果不想因為程式發生 index out of range 或由 panic 擲出的 error 導致整個程序強制中止的話可以用 recover 來攔截, 另外有時候程式的 error message 不夠明確時, 用 recover 有時候可以得到較明確的訊息

defer func() {
    if e := recover(); e != nil {
        // e is the interface{} typed-value we passed to panic()
        fmt.Println("Whoops: ", e) // Prints "Whoops: boom!"
    }
}()
var qq []string
fmt.Println(qq[0]) // 或直接用 panic("boom!")
fmt.Println("This will never be called")

recover() 要寫在最前面,否則會無法攔截到

將 stack 裡的東西印出來

func main() {
    defer func() {
        if e := recover(); e != nil {
            for i := 1; ; i++ {
                _, file, line, ok := runtime.Caller(i)
                if !ok {
                    break
                }
                log.Println(fmt.Sprintf("%s:%d", file, line))
            }
        }
    }()
    qq() // :21
    log.Println("Done!")
}

func qq() {
    ff() // :26
}

func ff() {
    panic("ff") // :30
}

Result:

2017/09/20 19:16:22 /usr/local/go/src/runtime/asm_amd64.s:514
2017/09/20 19:16:22 /usr/local/go/src/runtime/panic.go:489
2017/09/20 19:16:22 /tmp/qq.go:30
2017/09/20 19:16:22 /tmp/qq.go:26
2017/09/20 19:16:22 /tmp/qq.go:21
2017/09/20 19:16:22 /usr/local/go/src/runtime/proc.go:185
2017/09/20 19:16:22 /usr/local/go/src/runtime/asm_amd64.s:2197

其他

tag

type User struct {
    _    struct{} `type:"structure"`
    name string   `json:"name-field"`
    age  int
}

func main() {
    user := &User{name: "John Doe The Fourth", age: 20}

    if field, ok := reflect.TypeOf(user).Elem().FieldByName("_"); ok {
        fmt.Println(field.Tag)
    } else {
        panic("Field not found")
    }

    if field, ok := reflect.TypeOf(user).Elem().FieldByName("name"); ok {
        fmt.Println(field.Tag)
    } else {
        panic("Field not found")
    }
}

結果:

type:"structure"
json:"name-field"

runtime

  • The runtime library implements garbage collection, concurrency, stack management, and other critical features of the Go language.
  • The Go runtime manages scheduling, garbage collection, and the runtime environment for goroutines.

Pointer guildeline

  • struct call func by value 會複製自已一份, 如果 struct 很大的話用 pointer
  • 如果 func 裡面需要改變本身 struct 的值的話用 pointer
  • 有些 func 雖然不需要用 pointer, 但為了一致性還是使用 pointer 較好
  • Slices, maps and channels are reference types that do not require the extra indirection of an allocation with new
  • If the receiver is large, a big struct for instance, it will be much cheaper to use a pointer receiver.
  • If some of the methods of the type must have pointer receivers, the rest should too, so the method set is consistent regardless of how the type is used.
  • For types such as basic types, slices, and small structs, a value receiver is very cheap so unless the semantics of the method requires a pointer, a value receiver is efficient and clear.

盡量減少在其他子 package 用 var 定義全域變數

多使用全域來定義 struct, func, const,不得以才用 var 定義全域變數;如果有共用的 func 都會用到共同的變數,就用 struct 定義,否則容易產生 Memory Leak 的問題

go env

$ go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH=""
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
CC="gcc"
GOGCCFLAGS="-g -O2 -fPIC -m64 -pthread -fno-common"
CGO_ENABLED="1"

show variable type

import "reflect"
fmt.Println(reflect.TypeOf(tst))

判斷 interface{} 是否是 slice

if reflect.TypeOf(test["slice"]).Kind() == reflect.Slice {

}

debug

如同 PHP 的 print_r(), var_dump(), var_export(), 如果型態是 map 它也會把 key value 印出來

fmt.Printf("%v", whatever)

用上面的不會 struct 的欄位名稱,如果印出 struct 的欄位及值要用 :

fmt.Printf("%+v", whatever)

string literals are character sequences

string("Hello"[1])                  // ASCII only       e
string([]rune("Hello, 世界")[1])    // UTF-8            e
string([]rune("Hello, 世界")[8])    // UTF-8            界

指令

  • syntax checking : gofmt -e xx.go
  • 透過傳遞參數的方式改變 main.go 裡面 var 變數 : go build -ldflags '-X main.BUILD_ID=1'
  • go build 時太多 error 時你只會看到幾行然候其他就被省略成 too many errors, 改用 go build -gcflags="-e" 就可印出全部的 error

在 if 裡面 new 參數無法傳遞到外面

裡面有 new

var a bool
if true {
    a := true
    fmt.Println(a)
}
fmt.Println(a)

true
false

裡面沒 new

var a bool
if true {
    a = true
    fmt.Println(a)
}
fmt.Println(a)

true
true

MySQL NULL

null

string sql.NullString
int64 sql.NullInt64
float64 sql.NullFloat64

time 加上 tag

time.Time `sql:"default:null"`

或是用指標

string *string

for range 下在傳遞指標(pointer)需要注意

ii := []int{1, 2, 3}
dd := map[int]*int{}
dd2 := map[int]*int{}
for k, i := range ii {      // 不要使用 for range new 出來的 value, 因為最後都會指向同一個位置
    dd[k] = &i
}
for k, _ := range ii {      // 明確指定記憶體位址
    dd2[k] = &ii[k]
}
fmt.Println(dd)         // map[0:0x41602c 1:0x41602c 2:0x41602c]
fmt.Println(dd2)        // map[0:0x416020 1:0x416024 2:0x416028]

Golang 安裝

Linux / Mac

(最後更新: 2016-03-28)

[1] download golang source

Linux : https://storage.googleapis.com/golang/go1.6.linux-amd64.tar.gz

Mac : https://storage.googleapis.com/golang/go1.6.darwin-amd64.tar.gz

[2] Install

Linux : sudo tar -C /usr/local -xzf go1.6.darwin-amd64.tar.gz

Mac : sudo tar -C /usr/local -xzf go1.6.darwin-amd64.tar.gz

如果要重安裝直接 remove /usr/local/go 再執行上面指令就可以了

[3] Create essential folders

在家目錄下建立 mygo, 並且將 GOPATH 指向這裡, 這是 golang 一個很重要的設定, 可以將外部套件與原始程式分離, 你還必須在 mygo 下建立 3 個資料夾, src、pkg 及 bin

  • src 是放置外部套件(go get github.com/xx/cc 會載至這裡)及開發程式的地方
  • pkg 是程式編譯時產生的 .a 檔, 一般開發可以不用理會它
  • bin 是程式編譯後產生的 binary 執行檔

.bashrc(linux) / .bash_profile(mac) :

export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

Linux exec : source ~/.bashrc

Mac exec : source ~/.bash_profile

解壓縮完且 .bashrc 設定好以上就算安裝完成了, 可用 go version 測試是否安裝成功

Upgrade go version

  1. 移除 /usr/local/go
  2. 將官網載下來的 go 解壓縮到 /usr/local
  3. Done!
Mac upgrade go version 後還是顯示舊的版本

可能是曾經有用過 brew 安裝過 go, 即使你更新完 /usr/local/go version 還是顯示舊版的

使用 which go 可以發現它的指令是指到 /usr/local/bin/go 而它又是 link 到 /usr/local/Cellar/go/{舊版號}/bin/go

建議直接執行 brew uninstall go, 它會幫你把 /usr/local/Cellar/go 移除掉 (Uninstalling /usr/local/Cellar/go/1.7... (6,435 files, 250.6MB))

go version 後的版本就會是正常了

[註] 編譯

在本機編譯 :

go build

假如要編譯給 windows 的電腦執行 :

# 32 位元
GOOS=windows GOARCH=386 go build -o test.exe

# 64 位元
GOOS=windows GOARCH=amd64 go build -o test.exe

假如你的 project 有引入 C 程式則需要使用 CGO 來編譯 :

CGO_ENABLED=1 GOOS=windows GOARCH=amd64 go build -o test.exe

windows :

go 下載頁面 按照電腦的 OS 選擇你要下載的, 下載完直接安裝

安裝好 GO 後, 還需要 GCC 編譯環境, 因為 Mac 的 Xcode 本身就有 GCC, windows 沒有, 所以選擇了 mingw 這個輕量型的 GCC

參考安裝 MinGW


設定 GOPATH, 這是放一些你自己或 go get from github 存放的地方

  1. c:\Go 下建立 mygo 資料夾, 並且在 mygo 下建立三個資料夾 src, pkg, bin
  2. 設定環境變數

GOROOT : C:\Go GOPATH : C:\Go\mygo PATH : 在最後面加上 ;%GOROOT%\bin;%GOPATH%\bin;C:\Program Files (x86)\Git\bin

除了加上 GOROOT\bin、GOPATH\bin, 也可視需求加上 git bin 的路徑

重新啟動 cmd 使變數生效


編譯本機系統 GO 的編譯環境, 直接以滑鼠點擊執行 C:\Go\src\all.bat

跑完後, 建立自定義檔名的 .bat 檔, 如 C:\Go\src\jex.bat :

set CGO_ENABLED=0
set GOARCH=386
set GOOS=windows
call make.bat --no-clean
pause

雙擊它執行

然候在 C:\Go\bin 建立自定義檔名的 .bat 檔, 如 exec.bat :

set GOARCH=386
set GOOS=windows
go build

然候新增一個 hello.go 的檔案測試

package main
import (
    "fmt"
    "bufio"
    "os"
)
func main() {
    fmt.Printf("Hello, world\n")
    bufio.NewReader(os.Stdin).ReadBytes('\n') // 在 windows 結束後讓它停留在最後一行, 不要直接關掉
}

雙擊 exec.bat, 應該會在 C:\Go\bin 下出現 bin.exe 這個可執行檔

或者是在 cmd 到 c:\Go\bin, go build hello.go

執行它如果出現 Hello, world 恭喜你~設定完成!

Hello World!

hello.go :

package main

import "fmt"

func main() {
    fmt.Printf("Hello, world\n")
}

Package fmt implements formatted I/O with functions analogous to C’s printf and scanf. The format ‘verbs’ are derived from C’s but are simpler. Read detail

execute :

$ go run hello.go
Hello, world

編譯成執行檔

除了使用 go build 自動幫我們生成好執行檔, 我們也可以手動編譯及連結

ubuntu 手動編譯

/usr/lib/go/pkg/tool/linux_amd64/6g -o hello.6 hello.go
/usr/lib/go/pkg/tool/linux_amd64/6l -o hello hello.6
./hello

Mac 手動編譯

/usr/local/go/pkg/tool/darwin_amd64/6g -o hello.6 hello.go
/usr/local/go/pkg/tool/darwin_amd64/6l -o hello hello.6
./hello

編譯器

5g : ARM 6g : AMD 64 (x86-64) 8g : Intel 386 (X86)

386 : x86 架構的 32 位元 cpu 指令集, 簡稱 i386 AMD 64 : AMD 擴展 x86 的 64 位元指令集

[20140309] golang編譯器 :

  • 6g(X86-64)、8g(X86)、5g(arm) (及其支援工具,總稱為 gc)用 C 寫成,使用 yacc/Bison 作為解析器。Gccgo 是一個使用標準GCC作為後端的Go編譯器
  • 6g 是amd64的go編譯器,它生成的是 .6 檔案
  • 而 386 使用 8g 命令,它生成的一般是 .8 格式的檔案
  • 5g 是用於 arm 的 cpu,同理 amd64 用 6l,386 用 8l, arm 用 5l 的鏈接器!)

使用 vim 開發

必用套件vim-go

Golang - Native Package

fmt

輸出固定長度

fmt.Printf("ID : %-10s", id)    // 十個字元的長度, 向左對齊
fmt.Printf("ID : %10s", id)     // 十個字元的長度, 向右對齊
fmt.Printf("ID : %.10s", id)    // 印出頭十個字元
fmt.Printf("%q", []string{"a","b"}) // 印 slice `["a" "b"]`

前面補 0

fmt.Sprintf("%02d:%02d", 5, 3)  // 05:03

接收 command 輸入的值

fmt.Print("Enter a number: ")
var input float64
fmt.Scanf("%f", &input)

印出顏色

fmt.Println("\x1b[31;1mHello, World!\x1b[0m")

最簡單的方法, 不過有個缺點是它會是 string

fmt.Sprintf("%.7f", 123.123456789)                              // 123.1234568

不改變本身的型態

val := 123.123456789
pow10_n := math.Pow10(7)                                        // 1e7 = 10000000
d := math.Trunc(val*pow10_n) / pow10_n                          // 123.1234567

如果已知位數, 可簡化

math.Trunc(123.1234567*1e5) / 1e5                               // 123.12345

不改變本身的型態且4捨5入

val := 123.123456789
pow10_n := math.Pow10(7)
d := math.Trunc((val+0.5/pow10_n)*pow10_n) / pow10_n            // 123.1234568

log

儲存成 file

// open a file
f, err := os.OpenFile("dev.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0666)
if err != nil {
    log.Printf("Create log file error: %v", err)
}

// don't forget to close it
defer f.Close()

// assign it to the standard logger
log.SetOutput(f)

// 以下的 log 都會進到 log file 裡
log.Println("test")
log.Println("testr2")
log.Println("testr3")

將 log 內容暫存在 buffer

var buf bytes.Buffer
logger := log.New(&buf, "logger: ", log.Lshortfile)
logger.Print("Hello, log file!")

fmt.Print(&buf)

攔截 log 的內容

var buf bytes.Buffer
logger := log.New(&buf, "", log.Lshortfile)
go func() {
    for {
        if buf.Len() > 0 {
            fmt.Print(&buf)
            buf.Reset()
        }
    }
}()
fmt.Println(0)
logger.Print("Hello, log file!1")
logger.Print("Hello, log file!2")
logger.Print("Hello, log file!3")

dd := make(chan int)
<-dd

寫到 syslog

// Configure logger to write to the syslog. You could do this in init(), too.
logwriter, e := syslog.New(syslog.LOG_NOTICE, "myprog")
if e == nil {
    log.SetOutput(logwriter)
}

// Now from anywhere else in your program, you can use this:
log.Print("Hello Logs!")

/var/log/system.log :

Nov  4 08:19:33 Jexde-MacBook-Pro myprog[90449]: 2016/11/04 08:19:33 Hello Logs!

net

取得本機上的 Mac

interfaces, err := net.Interfaces()
if err != nil {
    panic("Poor soul, here is what you got: " + err.Error())
}
for _, inter := range interfaces {
    fmt.Println(inter.Name, inter.HardwareAddr)
}

header / status / etc.

header

resp, _ := http.Get(url)
// &{200 OK 200 HTTP/1.1 1 1 map[Connection:[keep-alive] Server:[nginx] Accept-Ranges:[bytes] Content-Length:[994291] Date:[Sat, 25 Jan 2014 19:13:51 GMT] Etag:["4e28a21e-f2bf3"] Content-Type:[video/x-flv] Last-Modified:[Thu, 21 Jul 2011 22:03:10 GMT]] 0xf8400c2a00 994291 [] false map[] 0xf8400c0240}

Status

resp.Status             // 200 OK
resp.StatusCode         // 200

size

resp.Header.Get("Content-Length")
// 994291

body

source, _ := ioutil.ReadAll(resp.Body)
// source 為 131 239 216 100 96 184 2 221 171 162 131 49 17 33 17 39 152 176 194 18 19 62 40 124 93 230 48 58 9 (..略..)

len(source) 結果會和 header 的 Content-Length 一樣

net/url 解析 URL

s := "postgres://user:pass@host.com:5432/path?k=v#f"
u, err := url.Parse(s)
fmt.Println(u.Scheme)                   // postgres
fmt.Println(u.User)                     // user:pass
fmt.Println(u.User.Username())          // user
p, _ := u.User.Password()
fmt.Println(p)                          // pass
fmt.Println(u.Host)                     // host.com:5432
h := strings.Split(u.Host, ":")
fmt.Println(h[0])                       // host.com
fmt.Println(h[1])                       // 5432
fmt.Println(u.Path)                     // /path
fmt.Println(u.Fragment)                 // f
fmt.Println(u.RawQuery)                 // k=v
m, _ := url.ParseQuery(u.RawQuery)
fmt.Println(m)                          // map[k:[v]]
fmt.Println(m["k"][0])                  // v

u, _ := url.Parse("http://www.test.com/xxx.mp3?foo=bar&foo=baz#this_is_fragment")
fmt.Println("full uri:", u.String())  // full uri: http://www.test.com/xxx.mp3?foo=bar&foo=baz#this_is_fragment
fmt.Println("scheme:", u.Scheme)      // scheme: http
fmt.Println("opaque:", u.Opaque)      // opaque:
fmt.Println("Host:", u.Host)          // Host: www.test.com
fmt.Println("Path", u.Path)           // Path /xxx.mp3
fmt.Println("Fragment", u.Fragment)   // Fragment this_is_fragment
fmt.Println("RawQuery", u.RawQuery)   // RawQuery foo=bar&foo=baz
fmt.Printf("query: %#v\n", u.Query()) // query: url.Values{"foo":[]string{"bar", "baz"}}

url 組 query

u, _ := url.Parse("https://example.com")
u.Path += "/api/v1/user/event"
query := url.Values{}
query.Add("a", "12+3")
query.Add("b", "4&D")
u.RawQuery = query.Encode()

output

example.com
/api/v1/user/event?a=12%2B3&b=4%26D
a=12%2B3&b=4%26D
https://example.com/api/v1/user/event?a=12%2B3&b=4%26D

It Urlencoded automatically.

已存在的 url 再新增 query param

var u *url.URL
u, _ = url.Parse("https://example.com/ccccc/ffff.jpg?a=aa&c=cc")
q := u.Query()
q.Set("q", "golang")
u.RawQuery = q.Encode()
fmt.Println(u.String())

自組 post body 要 escape

Post body 規則像 url, 要 url encode, 否則 + 等等的符號會被當成是 space

body := fmt.Sprintf("p=%s&iv=%s", url.QueryEscape(p_64), url.QueryEscape(iv_64))

net/http Download file

out, err := os.Create("output.txt")
defer out.Close()
resp, err := http.Get("http://example.com/")
defer resp.Body.Close()
n, err := io.Copy(out, resp.Body)

只取得 response 的 Header

res, _ := http.Head(url)
maps := res.Header

它連完線馬上就會斷了,不需要手動關掉

Get filename from url path

import (
    "net/url"
    "path/filepath"
)

s := "https://talks.golang.org/2012/splash/appenginegophercolor.jpg"
u, _ := url.Parse(s)
fmt.Println(filepath.Base(u.Path))

GET method

import (
  "http"
  "io/ioutil"
  "os"
)

response, _, err := http.Get("http://golang.org/")
if err != nil {
    fmt.Printf("%s", err)
    os.Exit(1)
} else {
    defer response.Body.Close()
    contents, err := ioutil.ReadAll(response.Body)
    if err != nil {
        fmt.Printf("%s", err)
        os.Exit(1)
    }
    fmt.Printf("%s\n", string(contents))
}

ref : https://gist.github.com/ijt/950790

Get 另一種寫法,指定 header

client := http.Client{
  Timeout: time.Second * 5,
}
req, err := http.NewRequest("GET", "http://www.google.com/dd", nil)
req.Close = true        // Note !!  避免發生 POST EOF 問題
req.Header.Set("Content-Type", "text/plain")
req.Header.Add("Content-Type", "text/plain")
resp, err := client.Do(req)
if err != nil {
    return err.Error()
}
defer resp.Body.Close()

if resp.StatusCode != http.StatusOK {
  err = errors.New("Server return non-200 status")
  return
}
contents, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(contents))
return

post :

http.NewRequest("GET", "http://www.google.com/dd", strings.NewReader("name=cjb"))

Post (form-data / json)

// New request
req, err := http.NewRequest("POST", c.URL, bytes.NewBuffer([]byte(c.PostData)))
req.Close = true
for k, v := range c.Header {
    req.Header.Set(k, v)
}

// 根據不同 Post type 加上對應的 header
switch strings.ToUpper(c.PostType) {
case "FORM":
    req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
case "JSON":
    req.Header.Set("Content-Type", "application/json")
}

// Send request
resp, err = client.Do(req)
if err != nil {
    return
}
defer resp.Body.Close()

Post (file)

// 模擬 form 的一個 選項, 現在是空的
bodyBuf := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(bodyBuf)

// 設定要上傳的檔案到 form 裡
fileWriter, err := bodyWriter.CreateFormFile(c.FileFieldName, c.FileFieldName)  // 一個是 file name, 一個是 field name, 都用 field name 就好了
if err != nil {
    err = errors.New("Failed to create form file")
    return
}

// 支援 file path 及 file bytes
var file *os.File
if c.FilePath != "" {
    // File path
    file, err = os.Open(c.FilePath)
    if err != nil {
        err = errors.New("Failed to open " + c.FilePath)
        return
    }
    defer file.Close()

    if _, err = io.Copy(fileWriter, file); err != nil {
        return
    }
} else {
    // Bytes
    r := bytes.NewReader(c.Bytes)
    if _, err = io.Copy(fileWriter, r); err != nil {
        return
    }
}

// 設定其他 form-data 的欄位跟值
// 它一定要在 bodyWriter.Close() 前, 因為這 close 完畢就是 boundary 的結尾, 有些在實作這功能時看到這個結尾就不會再繼續 parse 接下來 form-data 的欄位了 (Go Echo 上是這樣, 但 php 沒問題)
for key, val := range c.Params {
    if err = bodyWriter.WriteField(key, val); err != nil {
        return
    }
}

bodyWriter.Close()

// New request
req, err := http.NewRequest("POST", c.URL, bodyBuf)
// req.Close = true
for k, v := range c.Header {
    req.Header.Set(k, v)
}
req.Header.Set("Content-Type", bodyWriter.FormDataContentType()) // 取得上傳檔案的 Content-Type 的值, e.g. multipart/form-data; boundary=001e5fc50cf83d32c170bfef235709ba9016b0468c93ad2f108551a9f48c

// Send request
resp, err = client.Do(req)
if err != nil {
    return
}
defer resp.Body.Close()

Http 建立 API

http.HandleFunc("/", Index)
err := http.ListenAndServe(":5555", nil)
if err != nil {
    log.Fatal("ListenAndServe: ", err)
}

func Index(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, string("index"))
    // w.Write([]byte("Time: " + time.Now().Format(time.RFC1123)))
}

也可以改由這樣連線, 先建立 linstener

http.HandleFunc("/", Index)
s := &http.Server{}
l, err := net.Listen("tcp", ":5555")
if err != nil {
    panic(err)
}
panic(s.Serve(l))

ConnState 可以截取 connection 的狀態

總共有這五種狀態 StateNew StateActive StateIdle StateHijacked StateClosed

func main() {
    http.HandleFunc("/", myHandler)
    s := &http.Server{
        Addr:           ":8081",
        ReadTimeout:    10 * time.Second,
        WriteTimeout:   10 * time.Second,
        MaxHeaderBytes: 1 << 20,
        ConnState:      ConnStateListener,
    }
    panic(s.ListenAndServe())
}

func ConnStateListener(c net.Conn, cs http.ConnState) {
    fmt.Printf("CONN STATE: %v, %v\n", cs, c)
    // 如果想要直接結束 connection : c.Close()
}

ref : http://siddontang.com/2015/01/25/stop-server-gracefully/

handle connection (created at 27 Dec 2013)

ln, err := net.Listen("tcp", ":8080")
if err != nil {
    log.Fatal("Starting error", err)
}
conn, err := ln.Accept() // this blocks until connection or error
if err != nil {
    fmt.Println("Create connection failure!")
}
handleConnection(conn)

func handleConnection(conn net.Conn) {
    reader := bufio.NewReader(os.Stdin)
    for {
        fmt.Print("Please enter a message : ")
        str, err := reader.ReadString('\n')
        if err != nil {
            break
        }
        if str != "" {
            str := fmt.Sprintf("HTTP/1.1 200 OK\nContent-Length: %d\nContent-Type: text/html\n\n%s", len(str) - 1 , str)
            conn.Close() // shut down the connection
            break
        }
    }
    bufio.NewReader(os.Stdin).ReadBytes('\n')
}

TCP example

TCP server

func launchTCPserver() {
    // Listen for incoming connections.
    l, err := net.Listen("tcp", ":8081")
    if err != nil {
        fmt.Println("Error listening:", err.Error())
        os.Exit(1)
    }
    // Close the listener when the application closes.
    defer l.Close()
    fmt.Println("Listening on " + CONN_HOST + ":" + CONN_PORT)
    for {
        // Listen for an incoming connection.
        conn, err := l.Accept()
        if err != nil {
            fmt.Println("Error accepting: ", err.Error())
            os.Exit(1)
        }
        // Handle connections in a new goroutine.
        go handleRequest(conn)
    }
}

// Handles incoming requests.
func handleRequest(conn net.Conn) {
    OuterLoop:
        for {
            // will listen for message to process ending in newline (\n)
            message, err := bufio.NewReader(conn).ReadString('\n')
            switch err {
            case io.EOF:
                fmt.Printf("client disconnected: %v\n", err)
                break OuterLoop
            case nil:
            default:
                fmt.Println(err)
                break OuterLoop
            }
            // output message received
            fmt.Print("[server] Message Received:", string(message))
            // Processing
            time.Sleep(1 * time.Second)
            // send new string back to client
            conn.Write([]byte(string(message) + "\n"))
        }
}

TCP client

conn, _ := net.Dial("tcp", "127.0.0.1:8081")
for {
    // read in input from stdin
    reader := bufio.NewReader(os.Stdin)
    fmt.Println("-----")
    fmt.Print("Text to send: ")
    text, _ := reader.ReadString('\n')
    // send to socket
    fmt.Fprintf(conn, text+"\n")
    // listen for reply
    message, _ := bufio.NewReader(conn).ReadString('\n')
    fmt.Print("[Client] Message from server: " + message)
}

or

reply := make([]byte, 1024)
_, err = conn.Read(reply)
if err != nil {
    println("Write to server failed:", err.Error())
}
fmt.Println(string(reply))

API Custom Handler

type timeHandler struct {
  format string
}

func (th *timeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  tm := time.Now().Format(th.format)
  w.Write([]byte("The time is: " + tm))
}

func main() {
  mux := http.NewServeMux()

  th := &timeHandler{format: time.RFC1123}
  mux.Handle("/time", th)

  log.Println("Listening...")
  http.ListenAndServe(":3000", mux)
}

or

func (th *timeHandler) CustomName(w http.ResponseWriter, r *http.Request) {}

mux.HandleFunc("/time", th.CustomName)

net/http - x509: certificate signed by unknown authority

Post 到一個 https 的 URL 得到 x509: certificate signed by unknown authority

會發生這個原因主要是 golang 的 client 端會自動地對 server 傳來的 certificate 做驗證,但因為它是由不知名的 CA 簽發的,所以有 error

網路其中解法之一 : To fix this problem, you must add certificates of “COMODO RSA Domain Validation Secure Server CA” to your certififcate served from server, also see full report at https://www.ssllabs.com/ssltest/analyze.html?d=catchchat.catchchatchina.com

另一個解法是不要去驗證它

import ("net/http"; "crypto/tls")

tr := &http.Transport{
    TLSClientConfig: &tls.Config{InsecureSkipVerify : true},
}
client := &http.Client{Transport: tr}
resp, err := client.Get("https://someurl:443/)

更好的解法是對 server 的 certificate 做驗證, 這邊有點超越我的能力了, 請參考這

net/http client request 得到 EOF 原因

Post https://xxxxx.com/wait/: EOF

有可能是這個 request 需要比較久的時間,即使你的 http client 的 timeout 設很長, 也會因為 server timeout 太短先把你踢掉,所以你會得到這個 Error

Get lan ip

func GetLocalIP() string {
    addrs, err := net.InterfaceAddrs()
    if err != nil {
        return ""
    }
    for _, address := range addrs {
        // check the address type and if it is not a loopback the display it
        if ipnet, ok := address.(*net.IPNet); ok && !ipnet.IP.IsLoopback() {
            if ipnet.IP.To4() != nil {
                return ipnet.IP.String()
            }
        }
    }
    return ""
}

Proper way to test CIDR membership of an IP 4 or 6 address example

ref: https://www.socketloop.com/tutorials/golang-proper-way-to-test-cidr-membership-of-an-ip-4-or-6-address-example

package main

import (
        "fmt"
        "net"
        "os"
)

func main() {
        // generate a range of IP version 4 addresses from a Classless Inter-Domain Routing address
        ipAddress, ipNet, err := net.ParseCIDR("123.45.67.64/27")

        if err != nil {
                fmt.Println(err)
                os.Exit(1)
        }

        // generate a range of IPv4 addresses from the CIDR address
        var ipAddresses []string

        for ipAddress := ipAddress.Mask(ipNet.Mask); ipNet.Contains(ipAddress); inc(ipAddress) {
                //fmt.Println(ipAddress)
                ipAddresses = append(ipAddresses, ipAddress.String())
        }

        // list out the ipAddresses within range
        for key, ipAddress := range ipAddresses {
                fmt.Printf("[%v] %s\n", key, ipAddress)
        }

        //test for IP version 4 address for membership

        // WRONG WAY!!
        fmt.Println("Contains 123.45.67.69 : ", ipNet.Contains([]byte("123.45.67.69")))

        // CORRECT / PROPER WAY!
        fmt.Println("Contains 123.45.67.69 : ", ipNet.Contains(net.ParseIP("123.45.67.69")))

}

func inc(ip net.IP) {
        for j := len(ip) - 1; j >= 0; j-- {
                ip[j]++
                if ip[j] > 0 {
                        break
                }
        }
}

output :

[0] 123.45.67.64
[1] 123.45.67.65
...略...
[9] 123.45.67.73
...略...
[31] 123.45.67.95
Contains 123.45.67.69 :  false
Contains 123.45.67.69 :  true

io

pipe

pr, pw := io.Pipe()
go func() {
    defer pw.Close()
    for i := 1; i <= 3; i++ {
        _, err := fmt.Fprintf(pw, "Hello %d\n", i)
        if err != nil {
            panic(err)
        }
        time.Sleep(1 * time.Second)
    }
}()

_, err := io.Copy(os.Stdout, pr)
if err != nil {
    panic(err)
}

io/ioutil/bufio

將下載的資料轉成 io.Reader 型態

var source io.Reader
source = resp.Body
buffer := make([]byte, 1024)
for {
    cBytes, _ := source.Read(buffer)
    (...略...)

等同於

buffer := make([]byte, 1024)
for {
    cBytes,_ := resp.Body.Read(buffur)
    (...略...)

Read File

var img64 []byte
img64, _ = ioutil.ReadFile("/home/ubuntu/mygo/src/pushImage/google.png")

Read last line of file (ref: here)

func readLastLine(filename string) {
    var previousOffset int64 = 0

    file, err := os.Open(filename)
    if err != nil {
            panic(err)
    }

    defer file.Close()

    reader := bufio.NewReader(file)

    // we need to calculate the size of the last line for file.ReadAt(offset) to work

    // NOTE : not a very effective solution as we need to read
    // the entire file at least for 1 pass :(

    lastLineSize := 0

    for {
            line, _, err := reader.ReadLine()

            if err == io.EOF {
                    break
            }

            lastLineSize = len(line)
    }

    fileInfo, err := os.Stat(filename)

    // make a buffer size according to the lastLineSize
    buffer := make([]byte, lastLineSize)

    // +1 to compensate for the initial 0 byte of the line
    // otherwise, the initial character of the line will be missing

    // instead of reading the whole file into memory, we just read from certain offset

    offset := fileInfo.Size() - int64(lastLineSize+1)
    numRead, err := file.ReadAt(buffer, offset)

    if previousOffset != offset {

            // print out last line content
            buffer = buffer[:numRead]
            fmt.Printf("%s \n", buffer)

            previousOffset = offset
    }

}

Get last-modified time of file

// Get file list from dir
files, err := ioutil.ReadDir("/tmp")
if err != nil {
    return
}

for _, file := range files {
    // Skip dir
    if file.IsDir() {
        continue
    }

    fmt.Printf("%s %s\n", file.Name(), file.ModTime())
}

Line counter

func lineCounter(r io.Reader) (int, error) {
    buf := make([]byte, 32*1024)
    count := 0
    lineSep := []byte{'\n'}

    for {
        c, err := r.Read(buf)
        count += bytes.Count(buf[:c], lineSep)

        switch {
        case err == io.EOF:
            return count, nil

        case err != nil:
            return count, err
        }
    }
}

file, err := os.Open("/tmp/test.log")
if err != nil {
    return
}
defer file.Close()
count, err := lineCounter(file)

有效率的 monitor file 並取得最新內容 (類似 tail 指令)

  1. 先 new File
  2. 用兩個 int 變數記錄上一次及偵測到檔案變更的位置 (也就是檔案大小, 用 os.Stat 取得)
  3. 使用 fsnotify/fsnotify 偵測變更, 有變更就取最新內容 (用 file.ReadAt 在上一次的位置開始讀取, 讀取的長度用目前的size減上一次size

code:

for {
        select {
        case event := <-watcher.Events:
                if event.Op&fsnotify.Write == fsnotify.Write {
                        // log.Println("modified file:", event.Name)
                        info, err := file.Stat()
                        if err != nil {
                                log.Println("failed to get file size, err: ", err)
                                break
                        }
                        curr_size = info.Size()
                        last_line, err := readLastLine(file, prev_size, curr_size)
                        if err != nil {
                                log.Println("failed to get last line, err: ", err)
                                continue
                        }
                        prev_size = curr_size
                        if last_line != "" {
                                log.Println(last_line)
                        }
                }
        case err := <-watcher.Errors:
                log.Println("error:", err)
                return
        }
}

func readLastLine(file *os.File, prev int64, curr int64) (last_line string, err error) {
        // start reading from the end of the file
        buf := make([]byte, curr-prev)
        n, err := file.ReadAt(buf, prev)
        if err != nil && err != io.EOF {
                return "", err
        }
        return string(buf[:n]), nil
}

io - Read into []byte from another []byte

io.Copy (900MB, 1s)

b2 := bytes.NewBuffer(nil)
_, err = io.Copy(b2, bytes.NewReader(b))
if err != nil {
    panic(err)
}

ioutil.ReadAll (900MB 3s)

b3, err := ioutil.ReadAll(bytes.NewReader(b))
if err != nil {
    panic(err)
}

buffer read (900MB, 3s)

buf := make([]byte, 4*1024)
br := bufio.NewReader(bytes.NewReader(b))
b2 := bytes.NewBuffer(nil)
for {
    n, err := br.Read(buf)
    if err != nil {
        if err == io.EOF {
            break
        }
        panic(err)
    }
    if n == 0 {
        break
    }

    if _, err := b2.Write(buf[:n]); err != nil {
        panic(err)
    }
}

buffer pipe read (900MB, 4s)

pr, pw := io.Pipe()
go func() {
    defer pw.Close()
    buf := make([]byte, 4*1024)
    br := bufio.NewReader(bytes.NewReader(b))
    for {
        n, err := br.Read(buf)
        if err != nil {
            if err == io.EOF {
                break
            }
            panic(err)
        }
        if n == 0 {
            break
        }

        if _, err := pw.Write(buf[:n]); err != nil {
            panic(err)
        }
    }
}()

b2 := bytes.NewBuffer(nil)
_, err = io.Copy(b2, pr)
if err != nil {
    panic(err)
}

string

基本處理

String to Array

strings.Split("a/b/c",  "/")

Array to String (join)

s := []string{"a", "b", "c"}
fmt.Println(strings.Join(s, "/"))

os

外部參數

  • command : go run t.go dd
  • t.go : fmt.Println(os.Args[1])

“main.main” 函數並沒有返回值

如果想返回一個出錯信息,可用系統調用強制退出:

os.Exit(1)

# Stop
os.Exit(0)

檔案 / 目錄 是否存在

檔案或目錄是否存在

if _, err := os.Stat(filename); os.IsNotExist(err) {
    fmt.Printf("no such file or directory: %s", filename)
    return
}

檔案資訊

func GetFileInfo(path string) (isExistent bool, fileInfo os.FileInfo) {
    fileInfo, err := os.Stat(path)
    if err != nil {
        // no such file or dir
        return false, nil
    }
    if fileInfo.IsDir() {
        // it's a directory
        return false, nil
    }
    // it's a file
    return true, fileInfo
}

目錄是否存在

func DirExists(path string) (bool) {
    fileInfo, err := os.Stat(path)
    if err != nil {
        // no such file or dir
        return false
    }
    if fileInfo.IsDir() {
        // it's a directory
        return true
    }
    // it's a file
    return false
}

檔案是否存在

func FileExists(path string) (bool) {
    fileInfo, err := os.Stat(path)
    if err != nil {
        // no such file or dir
        return false
    }
    if fileInfo.IsDir() {
        // it's a directory
        return false
    }
    // it's a file
    return true
}

目前位置

dir, err := filepath.Abs(filepath.Dir(os.Args[0]))
if err != nil {
        log.Fatal(err)
}
fmt.Println(dir)

列出指定路徑下的檔案或目錄

files, err := ioutil.ReadDir("./")
if err != nil {
    log.Fatal(err)
}
for _, f := range files {
        fmt.Println(f.Name())
}

file - 指定寫入位置

file, _ := os.OpenFile("tt", os.O_RDWR|os.O_APPEND, 0660)
n, err := file.WriteAt([]byte("中Start"), 3)
if err != nil {
    fmt.Println(err.Error())
}
fmt.Println(n)

返回的 n 是寫入的字節大小, 中文字佔 3 btyes, 英文 1 bytes

檔案 tt 寫入後的結果 : ^@^@^@中Start

其中 ^@ 為空字元, 共有 3 組 ^@ 是因為 WriteAt 指定從 3 開始寫入,

但檔案位置是從 0 開始算, 所以會有 3 組 ^@

寫入改成 n, err := file.WriteAt([]byte("678"), 6)

結果為 : ^@^@^@中678rt

註 : ^@ 佔 1byte, 佔 3 bytes

Create folder

 import (
   //"fmt"
   "os"
   "path/filepath"
 )

 func main() {
   // create a TestDir directory on current working directory
   os.Mkdir("." + string(filepath.Separator) + "TestDir",0777)
 }

在 windows 下執行可執行檔(.exe)後視窗停留不關閉

引入 :

import (
    "bufio"
    "os"
)

最後一行執行 :

bufio.NewReader(os.Stdin).ReadBytes('\n')

當執行 .exe 檔後視窗就不會關閉了

或用另一種寫法

b := make([]byte, 1)
os.Stdin.Read(b)

不同作業系統取得不同 path separator (slash, 斜線)

string(filepath.Separator)

os/exec

取得執行中的 binary 路徑
[binary path]
os.Args[0]                      # /home/apps/go/src/test/test

[dir path]
filepath.Dir(os.Args[0])        # /home/apps/go/src/test
執行外部指令
output, err := exec.Command("git", "rev-parse", "HEAD").Output()
if err != nil {
    return "unknown"
}
fmt.Println(string(output))

output 最後會含一個 ascii: 10 (hex: 0A) 的換行字元, 它並不是 \n 字元

signal

接收中止訊號(kill signal)

當程式執行後會停在 <-sigc,如果收到 kill 指令則會繼續往下走

sigc := make(chan os.Signal, 1)
signal.Notify(sigc, os.Interrupt, os.Kill)
<-sigc
log.Println("Abort!")

發送 signal

重啟 signal

syscall.Kill(syscall.Getpid(), syscall.SIGHUP)

Kill process signal

syscall.Kill(syscall.Getpid(), syscall.SIGTERM)

time

time

Datetime

time.Now()                  // 2016-12-29 17:33:23.784617257 +0800 CST

Format

 time.Now().Format(time.RFC3339)                            // 2017-02-14T06:59:21Z   可以以 string 型態被解到 time.Time 型態
 time.Now().Local().Format("2006-01-02 15:04:05 +0800")     // 2014-06-30 14:23:38 +0800
 time.Now().Format("2006-01-02 15:04:05")                   // 2015-02-03 04:16:54
 time.Now().String()                                        // 2017-01-10 06:51:14.271079336 +0000 UTC
 time.Now().UTC().String()                                  // 轉成 UTC+0

Timestamp

time.Now().Unix()           // 1483004003

Convert timestamp to time

tm := time.Unix(1484032609, 0)

Convert string to time

today := "2017-01-10 16:57:28"
t, err := time.Parse("2006-01-02 15:04:05", today)

Convert string to time in timezone

// 將指定的時間跟 timezone 轉成 time.Time
l, _ := time.LoadLocation("Asia/Taipei")
t, _ := time.ParseInLocation("2006-01-02 15:04:05", "2019-01-02 17:04:43", l)
fmt.Println(t.In(time.UTC).Format("2006-01-02 15:04:05")) // 2019-01-02 09:04:43
fmt.Println(t.In(l).Format("2006-01-02 15:04:05"))        // 2019-01-02 17:04:43

UTC: time.LoadLocation(“UTC”)

Convert string to time in RFC3339

ts, err := time.Parse(time.RFC3339, src)

Convert time to time in timezone

t := time.Now()
loc, err := time.LoadLocation("Asia/Taipei")
fmt.Println(t.String())             // 2017-01-10 08:41:19.833220416 +0000 UTC
fmt.Println(t.In(loc))              // 2017-01-10 16:41:19.833220416 +0800 CST
fmt.Println(t.In(loc).Format("2006-01-02 15:04:05"))

year, month, day, hour, min, sec

year, month, day := t2.Date()
fmt.Printf("Date : [%d]year : [%d]month : [%d]day \n", year, month, day)
// Date : [2017]year : [1]month : [10]day

hr, min, sec := t2.Clock()
fmt.Printf("Clock : [%d]hour : [%d]minutes : [%d] seconds \n", hr, min, sec)
// Clock : [16]hour : [57]minutes : [28] seconds

first/last day of month

y, m, _ := time.Now().Date()
first := time.Date(y, m, 1, 0, 0, 0, 0, loc)
last := first.AddDate(0, 1, 0).Add(-time.Nanosecond)

Parse time 的非預期狀況

timeTest := "20:36"
timezone := "Asia/Tokyo"
loc, err := time.LoadLocation(timezone)
if err != nil {
    log.Fatal(err)
}
t, err := time.ParseInLocation("15:04", timeTest, loc)
if err != nil {
    log.Fatal(err)
}
fmt.Println(t.Format("15:04"))
fmt.Println(t.UTC().Format("15:04"))

在某些主機會是預期結果

20:36
11:36

某些會是:

20:36
11:17

似乎也不是 go 版本不同的問題, 不知道確切發生的原因, 但如果將指定的時間用完整一點可以解決此問題

timeTest := time.Now().Format("2006-01-02") + " " + "20:36"
timezone := "Asia/Tokyo"

loc, err := time.LoadLocation(timezone)
if err != nil {
    log.Fatal(err)
}
t, err := time.ParseInLocation("2006-01-02 15:04", timeTest, loc)
if err != nil {
    log.Fatal(err)
}
fmt.Println(t.Format("15:04"))
fmt.Println(t.UTC().Format("15:04"))

Comapare

After / Before

time1 := "2016-05-15 12:22:00"
time2 := "2017-05-15 12:22:00"
t1, err := time.Parse("2006-01-02 15:04:05", time1)
t2, err := time.Parse("2006-01-02 15:04:05", time2)
if err == nil && t1.Before(t2) {
    fmt.Println("true")
}

延遲

time.Sleep(3 * time.Second)

time.Sleep(500 * time.Millisecond)

speed := 500
time.Sleep(time.Duration(speed) * time.Millisecond)

時間相加/相減

兩時間相減

startTime := time.Now()                     // 2014-01-26 18:17:22.185125 +0800 CST
time.Sleep(3 * time.Second)
endTime := time.Now()                       // 2014-01-26 18:17:25.186307 +0800 CST

var durationTime time.Duration = endTime.Sub(startTime)
fmt.Println(durationTime.String())

加減(日時秒)

timein := time.Now().Add(time.Hour * time.Duration(1))      // Add 1 hour
timein := time.Now().Add(time.Minute * time.Duration(1))    // Add 1 minute
timein := time.Now().Add(time.Second * time.Duration(1))    // Add 1 second

// 減的話一樣是用 Add 但在 Duration 上加上負號
timein := time.Now().Add(time.Minute * time.Duration(-10))


then := time.Now().AddDate(0, 0, 1)     // +1 day
then := time.Now().AddDate(0, 0, -1)    // -1 day
then := time.Now().AddDate(1, 1, 0)     // +1 year and 1 month

timestamp plus 30 seconds

time.Now().Unix() + 30

Week

time.Now().Weekday()            // Tuesday
int(time.Now().Weekday())       // 2 (Day of the week)

相減算時間差

time_start := time.Now()
// do something
log.Printf("elasped time: %s", time.Since(time_start))

3 * time.Second

如果前面的 int 從 string 轉型,是會噴錯的,要改成 :

time.Duration(params["idle_timeout"].(int)) * time.Second

印出到小數的 timestamp/datetime (e.g. 2017-06-28 02:58:46.452 +0000 UTC)

func main() {
        fmt.Println(unixMilli(time.Unix(0, 123400000)))
        fmt.Println(unixMilli(time.Unix(0, 123500000)))
        m := makeTimestampMilli()
        fmt.Println(m)
        fmt.Println(time.Unix(m/1e3, (m%1e3)*int64(time.Millisecond)/int64(time.Nanosecond)))
}

func unixMilli(t time.Time) int64 {
        return t.Round(time.Millisecond).UnixNano() / (int64(time.Millisecond) / int64(time.Nanosecond))
}

func makeTimestampMilli() int64 {
        return unixMilli(time.Now())
}

Result:

123
124
1498618861845
2017-06-28 03:01:01.845 +0000 UTC

Timestamp with three decimal places

fmt.Sprintf("%f", float64(time.Now().UnixNano())/1000000000) // 1483004003.785

math/rand

產生 0~7 (每一次產生不一樣的變數)

rand.Seed(time.Now().UnixNano())
r := rand.Intn(8)

等於

r := rand.New(rand.NewSource(time.Now().UTC().UnixNano()))
f := r.Intn(8)

產生一串隨機數值 (ex: 7572000213564998818)

r := rand.New(rand.NewSource(time.Now().UTC().UnixNano()))
f := r.Int63()

Unique key

32 bytes HEX

key := make([]byte, 16)       // 16 會產生 32 個字元
_, err := rand.Read(key)
if err != nil {
    log.Fatal(err)
}
mk := fmt.Sprintf("%X", key)  //小寫
mk := fmt.Sprintf("%X", key)  // 大寫
fmt.Println(mk)

sync

pool 特性

  • 臨時暫存的 pool (有點像 cache 的感覺)
  • GC 會把 pool 清空, 也因為如此, 不要用 pool 來當 connection pool (TCP, DB, Redis, etc.)
  • A Pool is safe for use by multiple goroutines simultaneously.
  • 減少 GC 成本, 提高效能
  • a dynamically-sized store of temporary output buffers, 會根據壓力下自已變大縮小

pool 是會被GC的

p := &sync.Pool{
    New: func() interface{} {
        var i interface{}
        return i
    },
}
p.Put(1)
p.Put(2)

a := p.Get()
b := p.Get()
p.Put(a)
p.Put(b)
fmt.Println("Before GC:", a, b) // 1,2
runtime.GC()

a = p.Get()
b = p.Get()
p.Put(a)
p.Put(b)
fmt.Println("After GC: ", a, b) // nil, nil

result:

Before GC: 1 2
After GC:  <nil> <nil>

Pool (First In, First out)

var pool sync.Pool
type Item struct {
    Order string
}
v := pool.Get() // You get nothing if pool is empty.
if v == nil {
    v = &Item{Order: "first"}
}
pool.Put(v)                      // Put the first item
pool.Put(&Item{Order: "second"}) // Put the second item
q := pool.Get()                  // You'll get the first item
fmt.Println(q.(*Item).Order)     // "first"
q = pool.Get()                   // You'll get the second item
fmt.Println(q.(*Item).Order)     // "second"

Pool 乎不會比較快 (或許 example 寫的不夠複雜)

// Pool for our struct A
var pool *sync.Pool

// A dummy struct with a member
type A struct {
    Name string
}

// Func to init pool
func initPool() {
    pool = &sync.Pool{
        New: func() interface{} {
            return new(A)
        },
    }
}

// Main func
func main() {
    count := 100000
    fmt.Println("count: ", count)

    // Initializing pool
    initPool()
    for i := 1; i < 5; i++ {
        d := pool.New().(*A)
        d.Name = "Init00" + strconv.Itoa(i)
        // fmt.Printf("%d -> d.Name = %s\n", i, d.Name)
        pool.Put(d)
    }
    var wg sync.WaitGroup
    a := time.Now()
    for i := 0; i < count; i++ {
        wg.Add(1)
        go func(i int) {
            d := pool.Get().(*A)
            if d.Name == "" {
                d.Name = "NewXX" + strconv.Itoa(i)
            } else {
                d.Name = d.Name + "-" + strconv.Itoa(i)
            }
            _ = d
            // fmt.Printf("%d -> d.Name = %s\n", i, d.Name)
            pool.Put(d)
            defer wg.Done()
        }(i)
    }
    wg.Wait()
    fmt.Println("with pool: ", time.Since(a))

    var wg2 sync.WaitGroup
    b := time.Now()
    for i := 0; i < count; i++ {
        wg2.Add(1)
        go func(i int) {
            d := A{Name: "NewXX" + strconv.Itoa(i)}
            _ = d
            // fmt.Printf("%d -> d.Name = %s\n", i, d.Name)
            defer wg2.Done()
        }(i)
    }
    wg2.Wait()
    fmt.Println("without pool: ", time.Since(b))
}

result:

count:  100000
with pool:  150.17869ms
without pool:  40.754878ms

Sync WaitGroup

當同一時間同步做很多事,但希望全部 goroutine 做完事才進行下一步

package main

import (
    "fmt"
    "sync"
)

func main() {
    var wg sync.WaitGroup
    for q := 0; q < 10; q++ {
        wg.Add(1)               # flag
        go func(i int) {
            fmt.Println(i)
            defer wg.Done()     # 通報這個 goroutine 做完了
        }(q)
    }
    wg.Wait()                   # 程式會在這裡等全部 goroutine 做完
}

Mutex

a := 0
var wg sync.WaitGroup
for i := 0; i < 10000; i++ {
    wg.Add(1)
    go func() {
        a++
        defer wg.Done()
    }()
}
wg.Wait()
fmt.Printf("a = %d\n", a)     // a = 9222  < 10000 because of race condition

use -race to verify

 ==================
WARNING: DATA RACE
Read at 0x00c420084008 by goroutine 7:
  main.main.func1()
      /tmp/mutex.go:17 +0x3f

Previous write at 0x00c420084008 by goroutine 6:
  main.main.func1()
      /tmp/mutex.go:17 +0x58

Goroutine 7 (running) created at:
  main.main()
      /tmp/mutex.go:15 +0xfb

Goroutine 6 (finished) created at:
  main.main()
      /tmp/mutex.go:15 +0xfb
==================
a = 9978
Found 1 data race(s)
exit status 66

use lock

a := 0
var wg sync.WaitGroup
var l sync.Mutex
for i := 0; i < 10000; i++ {
    wg.Add(1)
    go func() {
        l.Lock()
        a++
        l.Unlock()
        defer wg.Done()
    }()
}
wg.Wait()
fmt.Printf("a = %d\n", a)      // a = 10000

encoding

組出 SOAP

package main

import "fmt"
import "encoding/xml"

type MyRespEnvelope struct {
    XMLName xml.Name
    Body    Body
}

type Body struct {
    XMLName     xml.Name
    GetResponse completeResponse `xml:"activationPack_completeResponse"`
}

type completeResponse struct {
    XMLName xml.Name `xml:"activationPack_completeResponse"`
    Id      string   `xml:"Id,attr"`
    MyVar   string   `xml:"activationPack_completeResult"`
}

func main() {

    Soap := []byte(`<?xml version="1.0" encoding="UTF-8"?>
                    <soap:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/">
                    <soap:Body>
                    <activationPack_completeResponse Id="http://tempuri.org/">
                    <activationPack_completeResult xsi:type="xsd:string">Active</activationPack_completeResult>
                    </activationPack_completeResponse>
                    </soap:Body>
                    </soap:Envelope>`)

    res := &MyRespEnvelope{}
    if err := xml.Unmarshal(Soap, res); err != nil {
        fmt.Println(err.Error())
    }

    var val completeResponse = res.Body.GetResponse
    fmt.Println(val.MyVar)
}

ref : http://play.golang.org/p/957GWzfdvN

Parse xml into struct

Person.xml :

<Person>
    <FullName>Grace R. Emlin</FullName>
    <Company>Example Inc.</Company>
    <Email where="home">
        <Addr>gre@example.com</Addr>
    </Email>
    <Email where='work'>
        <Addr>gre@work.com</Addr>
    </Email>
    <Group>
        <Value>Friends</Value>
        <Value>Squash</Value>
    </Group>
    <City>Hanga Roa</City>
    <State>Easter Island</State>
</Person>

code :

package main

import (
    "encoding/xml"
    "fmt"
    "io/ioutil"
    "os"
)

type Email struct {
    Where string `xml:"where,attr"`
    Addr  string
}

type Address struct {
    City, State string
}

type Result struct {
    XMLName xml.Name `xml:"Person"`
    Name    string   `xml:"FullName"`
    Phone   string
    Email   []Email
    Groups  []string `xml:"Group>Value"`
    Address
}

func main() {

    var v Result
    xmlFile1, err := ioutil.ReadFile("Person.xml")
    if err != nil {
        fmt.Println("Error opening file: ", err)
        return
    }

    err1 := xml.Unmarshal(xmlFile1, &v)
    if err1 != nil {
        fmt.Printf("error: %v", err)
        return
    }
    fmt.Println(v)

    // 用 streaming 形式解析, 較容易處理大數據的 xml
    xmlFile, err := os.Open("Person.xml")
    if err != nil {
        fmt.Println("Error opening file: ", err)
        return
    }
    defer xmlFile.Close()
    decoder := xml.NewDecoder(xmlFile)
    for {
        t, _ := decoder.Token()
        if t == nil {
            break
        }
        switch se := t.(type) {
        case xml.StartElement:
            if se.Name.Local == "Person" {
                var d Result
                decoder.DecodeElement(&d, &se)
                fmt.Println(d)
            }
        }
    }

    fmt.Println(v.XMLName)
    fmt.Printf("Name=%s \n", v.Name)
    fmt.Printf("Where=%s   Addr=%s  \n", v.Email[0].Where, v.Email[0].Addr)
    fmt.Printf("Where=%s   Addr=%s  \n", v.Email[1].Where, v.Email[1].Addr)
    fmt.Printf("Groups : %s,  %s\n", v.Groups[0], v.Groups[1])
    fmt.Printf("City=%s,  State=%s  \n", v.Address.City, v.Address.State)
}

ref : http://www.cnblogs.com/yuanershi/archive/2013/01/29/2881192.html

Parse xml into map - clbanning/mxj

var xml = `
    <Person>
        <FullName>Grace R. Emlin</FullName>
        <Company>Example Inc.</Company>
        <Email where="home">
            <Addr>gre@example.com</Addr>
        </Email>
        <Email where='work'>
            <Addr>gre@work.com</Addr>
        </Email>
        <Group>
            <Value>Friends</Value>
            <Value>Squash</Value>
        </Group>
        <City>Hanga Roa</City>
        <State>Easter Island</State>
    </Person>`
res, err := mxj.NewMapXml([]byte(xml))
if err != nil {
    fmt.Println(err)
}
fmt.Println(res)
fmt.Println(res["Person"].(interface{}).(map[string]interface{})["FullName"])
fmt.Println(res["Person"].(interface{}).(map[string]interface{})["Email"].([]interface{})[1].(map[string]interface{})["Addr"])

result :

map[Person:map[FullName:Grace R. Emlin Company:Example Inc. Email:[map[-where:home Addr:gre@example.com] map[Addr:gre@work.com -where:work]] Group:map[Value:[Friends Squash]] City:Hanga Roa State:Easter Island]]
Grace R. Emlin
gre@work.com

Parse json (struct, map)

方法1 : Json 放進預先定義好的 Struct 裡

type Test struct {
    JsonTag string `json:"field1"` // 利用 Tag mapping 到欄位名稱
    Field2  []struct {
        SubField1 string
        SubField2 string
    }
}

res := &Test{}
if err := json.Unmarshal([]byte(jsonString), res); err != nil {
    fmt.Println(err)
}
fmt.Println(res.JsonTag)
fmt.Println(res.Field2[0].SubField1)
fmt.Println(res.Field2[1].SubField2)

方法2 : 以 map 方式將 json 讀進來

var res2 map[string]interface{}
if err := json.Unmarshal([]byte(jsonString), &res2); err != nil {
    fmt.Println(err)
}
fmt.Println(res2["field1"])
fmt.Println(res2["field2"].([]interface{})[0].(map[string]interface{})["subField1"])
fmt.Println(res2["field2"].([]interface{})[1].(map[string]interface{})["subField2"])

json unmarshal 預設數字讓它是 int64 而不是 float64

json 字串裡面有 "visible_at":1483079819,但 unmarshal 完 int 變成 1.483079819e+09

解決方法 :

var data = `{
    "id": 12423434,
    "Name": "Fernando"
}`
d := json.NewDecoder(strings.NewReader(data))
d.UseNumber()
var x interface{}
if err := d.Decode(&x); err != nil {
    log.Fatal(err)
}
fmt.Printf("decoded to %#v\n", x)

base64 加解密

data := "abc123!?$*&()'-=@~"

sEnc := b64.StdEncoding.EncodeToString([]byte(data))
fmt.Println(sEnc)           # YWJjMTIzIT8kKiYoKSctPUB+

sDec, _ := b64.StdEncoding.DecodeString(sEnc)
fmt.Println(string(sDec))   # abc123!?$*&()'-=@~

POST base64 with form-data

body := fmt.Sprintf("p=%s&iv=%s", url.QueryEscape(crypt.Base64Encode(p_bytes)), url.QueryEscape(crypt.Base64Encode(iv_bytes)))

json.Marshal 不要 escape 某些字元

當欄位裡面有 url 時,json marshal 會自動地將某些字元(e.g. &) escape 為 \\u0026,為了避免此情形改用 :

type Search struct {
        Query string `json:"query"`
}
data := &Search{Query: "http://google.com/?q=stackoverflow&ie=UTF-8"}
buf := new(bytes.Buffer)
enc := json.NewEncoder(buf)
enc.SetEscapeHTML(false)
err = enc.Encode(data)
fmt.Println(string(buf.Bytes()))            // 注意 Encode 會在字尾加上 `\n`

[output]
    {"query":"http://google.com/?q=stackoverflow&ie=UTF-8"}

或用取代的方式將被脫逸的符號還原

b, err := json.Marshal(v)
b = bytes.Replace(b, []byte("\\u003c"), []byte("<"), -1)
b = bytes.Replace(b, []byte("\\u003e"), []byte(">"), -1)
b = bytes.Replace(b, []byte("\\u0026"), []byte("&"), -1)

crypto

md5

h := md5.New()
io.WriteString(h, "Hello World!")
fmt.Printf("%x", h.Sum(nil))

hasher := md5.New()
hasher.Write([]byte("test"))
fmt.Println(hex.EncodeToString(hasher.Sum(nil)))

h := md5.Sum([]byte("test"))
fmt.Println(hex.EncodeToString(h[:]))

sha456

方法1

hash := sha256.New()
hash.Write([]byte("test"))
b := hash.Sum(nil)  // byte
fmt.Println(hex.EncodeToString(b[:]))

方法2

b := sha256.Sum256([]byte("test"))
fmt.Println(hex.EncodeToString(b[:]))

AES CBC encrypt

plaintext 必須為 aes.Blocksize (16) 的倍數


import (
    "crypto/aes"
    "crypto/cipher"
    "crypto/rand"
    "encoding/base64"
    "fmt"
    "io"
)

func main() {
    key := []byte("1234567890123456")
    plaintext := pad([]byte("secret data12345"))

    if len(plaintext)%aes.BlockSize != 0 {
        panic("plaintext is not a multiple of the block size")
    }

    block, err := aes.NewCipher(key)
    if err != nil {
        panic(err)
    }

    ciphertext := make([]byte, aes.BlockSize+len(plaintext))
    iv := ciphertext[:aes.BlockSize]
    if _, err := io.ReadFull(rand.Reader, iv); err != nil {
        panic(err)
    }

    mode := cipher.NewCBCEncrypter(block, iv)
    mode.CryptBlocks(ciphertext[aes.BlockSize:], plaintext)
    fmt.Println(base64.StdEncoding.EncodeToString(iv))              // iv base64: VGnqCX2VQonnNUL/Hlmk1w==
    fmt.Println(base64.StdEncoding.EncodeToString(ciphertext[16:])) // encrypt base64: 5ThQKq6RnAiGsLHU/BWm7A==
}

// plaintext 無法被 aes.Blocksize (16) 要先補位數湊滿
func pad(in []byte) []byte {
    padding := aes.BlockSize - (len(in) % aes.BlockSize)
    if padding == 0 {
        padding = aes.BlockSize
    }
    for i := 0; i < padding; i++ {
        in = append(in, byte(padding))
    }
    return in
}

另種寫法

text := []byte(`secret data12345`)
sharekey := []byte(`1234567890123456`)
block, err := aes.NewCipher(sharekey)
if err != nil {
    panic(errors.New("AESEncrypt Create Cipher error: " + err.Error()))
}

paddingText := pad(text, block.BlockSize())
iv := make([]byte, aes.BlockSize)
if _, err := io.ReadFull(rand.Reader, iv); err != nil {
    panic(err)
}

mode := cipher.NewCBCEncrypter(block, iv)
mode.CryptBlocks(paddingText, paddingText)
fmt.Println(base64.StdEncoding.EncodeToString(iv))
fmt.Println(base64.StdEncoding.EncodeToString(paddingText))

// 補滿 16 位
func pad(src []byte, blocksize int) []byte {
    pdSize := blocksize - (len(src) % blocksize)
    padBytes := bytes.Repeat([]byte{0x00}, pdSize)
    src = append(src, padBytes...)
    return src
}

AES CBC decrypt

import (
    "crypto/aes"
    "crypto/cipher"
    "encoding/base64"
    "fmt"
)

func main() {
    key := []byte("1234567890123456")
    iv, _ := base64.StdEncoding.DecodeString("7/JbVL/5cMvqf9sslD6qdQ==")                // iv base64
    ciphertext, _ := base64.StdEncoding.DecodeString("6AFagV3iLEAqbBuSqvL19Q==")        // encrypt base64

    block, _ := aes.NewCipher(key)
    if len(ciphertext) < aes.BlockSize {
        panic("ciphertext too short")
    }
    if len(ciphertext)%aes.BlockSize != 0 {
        panic("ciphertext is not a multiple of the block size")
    }
    aes := cipher.NewCBCDecrypter(block, iv)
    aes.CryptBlocks(ciphertext, ciphertext)
    fmt.Printf("%s\n", ciphertext)
}

AES Decrypt - String 相同, 同長度及 byte 不同 invalid character '\x00' after top-level value

可能是 AES 加解密後發生的情況,可能在過程中被塞入 padding, 導致解出來後的 byte 數量不一樣

加密前 :

2561953d9ec389715498d44b0c150fec
len = 32
[50 53 54 49 57 53 51 100 57 101 99 51 56 57 55 49 53 52 57 56 100 52 52 98 48 99 49 53 48 102 101 99]

加密後 :

len = 48
[50 53 54 49 57 53 51 100 57 101 99 51 56 57 55 49 53 52 57 56 100 52 52 98 48 99 49 53 48 102 101 99 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

ASCII 對應

50 => 2
99 => c
0 => 空字元(Null)

解決方法: 將多餘的 \x00 刪除

b = bytes.Trim(b, "\x00")

ref: 類似問題

UUID (crypto/rand)

// 一般
uuid := make([]byte, 16)
io.ReadFull(rand.Reader, uuid)

// 64 encode
uuid := Gen()
ret := base64.URLEncoding.EncodeToString(uuid)
ret = strings.Replace(ret, "=", "", -1)

import uuid "github.com/satori/go.uuid"
uuid.NewV4()

defer

關於 defer 如何運作

package main

import (
    "fmt"
)

func main() {
    defer Function3()
    Function1()
}

func Function1() {
    fmt.Println("Function1 開始")
    defer Function2()
    fmt.Println("Function1 結束")
}

func Function2() {
    fmt.Println("Function 2")
}

func Function3() {
    fmt.Println("Function 3")
}


執行結果 :

Function1 開始
Function1 結束
Function 2
Function 3

ref : http://ithelp.ithome.com.tw/question/10153473

error

擲出自定義錯誤訊息

err = errors.New("fail")

比對 error

在另一個 package 用全域變數定義

package fruits

var NoMorePumpkins = errors.New("No more pumpkins")

就可以比對了

package shop

if err == fruits.NoMorePumpkins {
     ...
}

runtime

detect OS

switch runtime.GOOS {
    case "windows":
    case "darwin":       //Mac OS
    case "linux":
    default :
}

記憶體使用量

var m0 runtime.MemStats
runtime.ReadMemStats(&m0)
fmt.Printf("Memory: %.2f mb", float64(m0.Sys)/1024/1024)

channel

把 Channel 當成 Queue 使用

它是先進先出,如果要做一個 worker 很好用,直接宣告它的大小是多少,但塞值不需要管它的 index,一直丟就對了

var worker_ch = make(chan string, 10000)

// 用 Goroutine 跑,一個 Worker,它會一直 wait,直到 worker_ch 有值
for {
    file_name := <-worker_ch
    DoSomething(file_name)
}

// 另一個 Goroutine,假設它是 API 負責塞值給 worker_ch
func NonBlcok(ctx *fasthttp.RequestCtx, _ fasthttprouter.Params) {
    file_name := string(ctx.FormValue("filename"))
    worker_ch <- file_name
}

用 channel & select 實作 long lived, concurrent safe pools

// Pool holds Clients.
type Pool struct {
    pool   chan *Client
}

// NewPool creates a new pool of Clients.
func NewPool(max int) *Pool {
    return &Pool{
        pool:   make(chan *Client, max),
    }
}

// Borrow a Client from the pool.
func (p *Pool) Borrow() *Client {
    var c *Client
    select {
    case c = <-p.pool:
    default:
        c = newClient()
    }
    return c
}

// Return returns a Client to the pool.
func (p *Pool) Return(c *Client) {
    select {
    case p.pool <- c:
    default:
        // let it go, let it go...
    }
}

reflect

Call dynamic func by name

func Call(m map[string]interface{}, name string, params ...interface{}) (result []reflect.Value, err error) {
    f := reflect.ValueOf(m[name])
    if len(params) != f.Type().NumIn() {
        err = errors.New("The number of params is not adapted.")
        return
    }
    in := make([]reflect.Value, len(params))
    for k, param := range params {
        in[k] = reflect.ValueOf(param)
    }
    result = f.Call(in)
    return
}

用法 :

// 宣告
funcs := map[string]interface{}{
    "qq": curl.Get,  // index 隨便取, 後面的是 func name  e.g. func QQ() 就填  QQ, 如果是 curl pacakge 的 Get func 就填 curl.Get
}
Call(funcs, "qq", params)  // 如果有參數就傳入 params

Call dynamic struct.function by name (string)

import "fmt"
import "reflect"

type T struct{}

func (t *T) Test() string {
    fmt.Println("test...")
    return "ok"
}

func main() {
    var t T
    res := reflect.ValueOf(&t).MethodByName("Test").Call([]reflect.Value{})
    fmt.Println(res[0]) // ok
}

res 是一個陣列,即使你的 func 只回傳一個參數,它也是放在陣列裡

flag

auto flag

config := struct {
    Name    string        `flag:"queue,queue name"`
}{
    Name:    "",
}
autoflags.Define(&config)
flag.Parse()

image

Get image’s width and height

import (
    "fmt"
    "image"
    _ "image/jpeg"
    "io/ioutil"
    "os"
    "path/filepath"
)

const dir_to_scan string = "/tmp/images"

func main() {
    files, _ := ioutil.ReadDir(dir_to_scan)
    for _, imgFile := range files {

        if reader, err := os.Open(filepath.Join(dir_to_scan, imgFile.Name())); err == nil {
            defer reader.Close()
            im, _, err := image.DecodeConfig(reader)
            if err != nil {
                fmt.Fprintf(os.Stderr, "%s: %v\n", imgFile.Name(), err)
                continue
            }
            fmt.Printf("%s %d %d\n", imgFile.Name(), im.Width, im.Height)
        } else {
            fmt.Println("Impossible to open the file:", err)
        }
    }
}

image/jpg Convert jpg to png

file, err := os.Open("test.jpg")
if err != nil {
    log.Fatal("File error")
}

img, err := jpeg.Decode(file)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}

out, err := os.Create("test.png")
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
err = png.Encode(out, img)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
fmt.Println("Done!")

image/jpg Convert jpg to bmp

// Convert jpg to bmp
img_file, err := os.Open("test.jpg")
if err != nil {
    log.Fatal("File error")
}
defer img_file.Close()

img, err := jpeg.Decode(img_file)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
out_file, err := os.Create("test.bmp")
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
err = bmp.Encode(out_file, img)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
defer out_file.Close()

image/jpg Convert bmp to jpg

// Convert jpg to bmp
img_file, err := os.Open("test.bmp")
if err != nil {
    log.Fatal("File error")
}
defer img_file.Close()

img, err := bmp.Decode(img_file)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
out_file, err := os.Create("test.jpg")
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}

options := &jpeg.Options{Quality: 50}
err = jpeg.Encode(out_file, img, options)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
defer out_file.Close()

convert an image to a black and white image

import (
    "fmt"
    "image"
    "image/draw"
    "image/jpeg"
    "os"
)

func main() {
    file, err := os.Create("test-result.jpg")
    if err != nil {
        fmt.Println(err)
    }
    defer file.Close()

    file1, err := os.Open("test.jpg")
    if err != nil {
        fmt.Println(err)
    }
    defer file1.Close()
    img, _ := jpeg.Decode(file1)

    jpg := image.NewGray(img.Bounds()) //NewGray

    draw.Draw(jpg, jpg.Bounds(), img, img.Bounds().Min, draw.Src)

    jpeg.Encode(file, jpg, nil)

}

Draw a line on an image

import (
    "fmt"
    "image"
    "image/color"
    "image/draw"
    "image/jpeg"
    "os"
)

func main() {
    file, err := os.Create("test2.jpg")
    if err != nil {
        fmt.Println(err)
    }
    defer file.Close()

    file1, err := os.Open("test.jpg")
    if err != nil {
        fmt.Println(err)
    }
    defer file1.Close()
    img, _ := jpeg.Decode(file1)

    // Create an image with size same as the original image
    jpg := image.NewRGBA(img.Bounds())    // or you can specify the size: image.NewRGBA(image.Rect(0, 0, 1080, 1080))

    // Put the original image into to the image that we just created
    draw.Draw(jpg, jpg.Bounds(), img, img.Bounds().Min, draw.Over)

    // Draw a red dot at (2, 3)
    for x := 1; x < 100; x++ {
        y := x
        jpg.Set(x, y, color.RGBA{255, 0, 0, 255})
    }

    // jpeg.Encode(file, jpg, nil)
    jpeg.Encode(file, jpg, &jpeg.Options{Quality: 100})
}

unsafe

看變數佔多數記憶體 (bytes)

a1 := "xxxdflasjdfl;daskjfdsalfkjasdlfjas"
a2 := int64(66666)
a3 := int32(5555)
a4 := float32(2222.4)
fmt.Println(unsafe.Sizeof(a1))  // 16
fmt.Println(unsafe.Sizeof(a2))  // 8
fmt.Println(unsafe.Sizeof(a3))  // 4
fmt.Println(unsafe.Sizeof(a4))  // 4

utf8

Count size of UTF8 string

utf8.RuneCountInString("世界")

reflect

Implementation of generics

type Animal interface{}

type Body struct {
    FeetCount int64
}

type People struct {
    Name string
    Body
}

type Cat struct {
    Name string
    Body
}

func main() {
    var h People
    var c Cat
    setName(&h, "People") // Must be passed by pointer
    setFeetCount(&h, 2)
    fmt.Println(h)

    setName(&c, "Cat")
    setFeetCount(&c, 4)
    fmt.Println(c)
}

// Set string
func setName(a Animal, name string) {
    o := reflect.ValueOf(a).Elem().FieldByName("Name")
    if o.CanSet() {
        reflect.ValueOf(a).Elem().FieldByName("Name").SetString(name)
        fmt.Println(reflect.ValueOf(a).Elem().FieldByName("Name").String())
    }
}

// Set struct
func setFeetCount(a Animal, c int64) {
    o := reflect.ValueOf(a).Elem().FieldByName("Body")
    if o.CanSet() {
        o.FieldByName("FeetCount").SetInt(c)
    }
}

output:

People
{People {2}}
Cat
{Cat {4}}

DHCP 運作流程

發送流程

[1] client 以 UDP 發出 DHCPDISCOVER 廣播封包尋找 DHCP Server

來源位址 : 0.0.0.0:68 目的位址 : 255.255.255.255:67

當 client 開機或重啟網卡時會發出此的封包

[2] DHCP Server 發出 DHCPOFFER 廣播封包給 client

來源位址 : 192.168.181.30:67 目的位址 : 255.255.255.255:68

收到請求會先依據 client 的 MAC 來判斷以下

  • 到伺服器的登錄檔中尋找該用戶之前是否曾經用過某個 IP ,若有且該 IP 目前無人使用,則提供此 IP 給用戶端;
  • 若設定檔針對該 MAC 提供額外的固定 IP (static IP) 時,則提供該固定 IP 給用戶端;
  • 若不符合上述兩個條件,則隨機取用目前沒有被使用的 IP 參數給用戶端,並記錄下來。

接著 server 提供一組 IP 及租約期限訊息給 client, 如果網路有很多台 DHCP Server, client 會以收到的第一個 DHCPOFFER 訊息為主

[3] client 發出 DHCPREQUEST 給 DHCP Server

來源位址 : 0.0.0.0:68 目的位址 : 255.255.255.255:67

Client 端在收到 DHCPOFFER 封包後,會先送出一個 ARP Request 的廣播封包,來確定對方所提供的 IP 位址沒有其他的主機在使用,

如果發現這個位址已被其它主機使用掉的話,Client 端就會送出一個 DHCPDECLINE 的拒絕訊息給 Server,然後再送出一個 DHCPDISCOVER 的廣播封包來重新索取 IP 位址。

事實上﹐並不是所有 DHCP 客戶端都會無條件接受 DHCP 伺服器的 offer ﹐尤其這些主機安裝有其它 TCP/IP 相關的客戶軟體。客戶端也可以用 Dhcprequest 向伺服器提出 DHCP 選擇

假設 client 確定此位址尚未被使用而且也接受這個 IP 的話, 那麼接著就會送出一個 DHCPREQUEST 的廣播封包來回應給這台 DHCP Server。

之所以還是送出廣播封包的用意,是為了告知其他未被選上的 DHCP Server,這樣這些 DHCP Server 才可將原本打算保留給 Client 的 IP 位址釋放出來給其他 Client 使用。

ARP(Address Resolution Protocol) : 從 cache 尋找 IP 位址所對應的 MAC

[4] DHCP Server 發出 DHCPACK 給 client

來源位址 : 192.168.181.30:67 目的位址 : 255.255.255.255:68

DHCP Server 收到 DHCPREQUEST 的訊息後,會回應一個 DHCPACK 的確認訊息給 Client ( 此時還是透過廣播方式,因 Client 尚未正式取得 IP 位址 ), 以確認 IP 租約的正式生效

此訊息裡包含了指派給 Client 端使用的 IP 位址、網路遮罩、預設閘道器、DNS Server 等等,

而在 client 收到這些訊息後,便完成了整個索取 IP 位址的動作。

IP 租約到期的情況 :

用戶端離線:不論是關閉網路介面 (ifdown)、重新開機 (reboot)、關機 (shutdown) 等行為,皆算是離線狀態,這個時候 Server 端就會將該 IP 回收,並放到 Server 自己的備用區中,等待未來的使用;

用戶端租約到期:前面提到 DHCP server 端發放的 IP 有使用的期限,用戶端使用這個 IP 到達期限規定的時間,而且沒有重新提出 DHCP 的申請時,就需要將 IP 繳回去!這個時候就會造成斷線。但用戶也可以再向 DHCP 伺服器要求再次分配 IP 囉。

其他

更新租約期:

往往 DHCP Client 為了持續延用此 IP 位址,而會向 DHCP Server 要求更新租約期,一旦更新成功,則 Client 的 IP 租約期限就會從頭開始計算起。在 Client 更新租約期的時候,會向 Server 送出一個 DHCPREQUEST 的訊息。接著以下就舉兩種 Client 送出 DHCPREQUEST 給 Server 的時機:

  1. DHCP Client 重新開機時:

當 DHCP 的用戶端電腦重新啟動時,會送出 DHCPREQUEST 給 DHCP Server,來要求延用原來的 IP 位址,倘若此位址目前尚未租用給其它的 Client,則用戶端就可以取得與上次所使用的 IP 相同。

  1. 達 ½ 租約期時:

DHCP Client 會在 IP 租約期達到一半時,主動送出一個 DHCPREQUEST 給 DHCP Server,而 Client 如果也順利收到對方所回應的 DHCPACK 訊息,就表示已成功更新租約期。萬一更新失敗,Client 還是可以繼續使用此位址,因為租約期還沒到。而在更新失敗的情況下,Client 會在租約期到了八分之七時,利用廣播方式送出 DHCPREQUEST 更新訊息來尋找任何一台可用的 DHCP Server,看看能否更新租約,如果還是無法更新成功,則 Client 會放棄目前使用的 IP,並重新送出 DHCPDISCOVER 來向 DHCP 索取一個新的位址。

檢查租約檔案

cat /var/lib/dhcpd/dhcpd.leases

ref : http://blog.roodo.com/dabinn/archives/7009437.html http://www.study-area.org/network/network_ip_dhcp.htm http://paching.myweb.hinet.net/lesson13.htm http://linux.vbird.org/linux_server/0340dhcp.php

Mkfifo 指令

介紹

用在兩個不同執行的程式溝通, 他就像是一條水管(pipeline file)將兩個程式連接起來

其中一邊不斷的將水倒進去, 另一邊則取水, 這樣就可以達到資料

Example

session 1 :

建立 pipe

cd /tmp
mkfifo test

傳送資料進去

echo "Hello Jex" > test
echo "line 1" > test&

在命令的最後面加上 & 可以讓它背景作業

session 2 :

將 session 1 存的資料拿出來

cd /tmp
cat test

fifo (命名通道)

這些參數似乎要用 C 寫才能使用

讀 :

  • O_NONBLOCK enable : 立刻返回成功
  • O_NONBLOCK dieable : block 直到有資料寫入而開啟 FIFO

寫 :

  • O_NONBLOCK enable : 立刻返回失敗, error code 為 ENXIO
  • O_NONBLOCK dieable : block 直到有資料寫入而開啟 FIFO

打開檔案預設是 block

mkfifo 與 fifo 的差異

mkfifo 由 pipe 函數建立並打開, fifo 由 mkfifo 函數建立, 打開用 open

差別只在建立與開啟的方式不同, 一旦完成這些, 在通道 I/O 及 FIFO 幾乎一樣

  • mkfifo - make FIFOs (named pipes)
  • mknod - make block or character special files

ref :

Nc (Netcat) 指令

介紹

用來連接特定 port, 或啟動一個 port 來監聽, 可以執行簡單的訊息或檔案傳送

有些系統將 nc 改為 netcat

快速看如何使用

啟動 20000 port

nc -l 192.168.1.171 20000
  • 如果不加上 IP, 就會監聽所有從 20000 port 進來的主機
  • -l :作為監聽之用,亦即開啟一個 port 來監聽用戶的連線;
  • -u :不使用 TCP 而是使用 UDP 作為連線的封包狀態

連線到 20000 port

nc 192.168.1.171 20000

然候就可以隨便打一些字, 送出後主機那就會看到了, 也可搭配 stdout, stdin 做互動輸入/輸出

傳送/接收 檔案

傳送檔案

$ cat backup.iso | nc 54.250.122.78 3333

接收檔案並存入 backup.iso

$ nc -l 3333 >> backup.iso

傳送顯示進度條

$ cat backup.iso | pv -b | nc 54.250.122.78 3333

接收顯示進度條

$ nc -l 8000 | pv -b >> backup.iso
  • pv can be used to show a progress indicator.
  • pv : Pipe Viewer, 顯示通過 pipe 的資料處理進度, 例如應用在 copy, compress, nc 等等..

傳送 partition

建立一個 partition 的 image 並傳送到遠端主機

$ dd if=/dev/hdb5 | gzip -9 | nc -l 3333

-9, –best : These options change the compression level used, with the -1 option being the fastest, with less compression, and the -9 option being the slowest, with optimal compression. The default compression level is 6.

連到傳送端的 server 並且接收 partition image

$ nc 192.168.0.1 3333 | pv -b > myhdb5partition.img.gz

傳送資料夾

壓縮檔案並傳到遠端主機

$ tar -czf - /etc/ | nc -l 3333

接收檔案

$ nc 192.168.0.1 3333 | pv -b > mybackup.tar.gz

使用 ssh 傳送檔案

有兩個好處 :

  1. 資料透過加密的通道傳送更為安全
  2. 你不需要在額外開 port, 因為使用的是 ssh port

You pipe the file to a listening socket on the server machine in the same way as before. It is assumed that an SSH server runs on this machine too. listen 一個 port 並以 pipe 傳送檔案 (假設 SSH server 已經在這台啟動了)

$ cat backup.iso | nc -l 3333

On the client machine connect to the listening socket through an SSH tunnel: client server 以 SSH 連到傳送端的主機

$ ssh -f -L 23333:127.0.0.1:3333 me@192.168.0.1 sleep 10; \
                nc 127.0.0.1 23333 | pv -b > backup.iso
  • 這個方式可以設定當檔案傳送完後自動關閉連線
  • ssh -f : Requests ssh to go to background just before command execution. This is useful if ssh is going to ask for passwords or passphrases, but the user wants it in the background.
  • ssh -L [bind_address:]port:host:hostport : Specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side. This works by allocating a socket to listen to port on the local side, optionally bound to the specified bind_address.

監聽一個範圍 port 的 connection

可以當作 Port Scanner

使用 -z, netcat 不會 initiate 一個連線到 server, 只會通知這個 port 有找到, 它可以允許 port-range 去掃

例如掃 80~90 的 port, 發送 80 port 有打開 :

$ nc -z 192.168.0.1 80-90
Connection to 192.168.0.1 80 port [tcp/http] succeeded!

ref : http://linux.vbird.org/linux_server/0140networkcommand.php#nc http://www.g-loaded.eu/2006/11/06/netcat-a-couple-of-useful-examples/

解析 TCP 三向交握(Three-way Handshake)封包

瀏覽器向 server 發送請求 :

瀏覽器發送 : 指定 seq

22:39:13.832224 IP 42.79.253.189.29729 > 172.31.23.23.8080: Flags [S], seq 3605791234, win 65535, options [mss 1394,nop,wscale 4,nop,nop,TS val 401990828 ecr 0,sackOK,eol], length 0

server 回應 : 將 seq+1 放到 ack, 並且指定 seq

22:39:13.832271 IP 172.31.23.23.8080 > 42.79.253.189.29729: Flags [S.], seq 4067821211, ack 3605791235, win 28960, options [mss 1460,sackOK,TS val 333436167 ecr 401990828,nop,wscale 7], length 0

瀏覽器發送 : ack = 1 用來回應上一個封包並且確認封包正確

22:39:13.943021 IP 42.79.253.189.29729 > 172.31.23.23.8080: Flags [.], ack 1, win 8205, options [nop,nop,TS val 401990926 ecr 333436167], length 0

瀏覽器發送 : Push 表示此封包應立即傳送不需等緩衝區滿了才送, seq : 1:321 代表從 1 開始, 並且加上長度 320 的 header

22:39:13.943692 IP 42.79.253.189.29729 > 172.31.23.23.8080: Flags [P.], seq 1:321, ack 1, win 8205, options [nop,nop,TS val 401990927 ecr 333436167], length 320

Browser request headers (320 bytes of data) :


GET / HTTP/1.1
Host: 54.250.138.78:8080
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-tw
Connection: keep-alive
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71

server 回應 : ack : 321 表示收到剛剛瀏覽器傳來的資料

22:39:13.943711 IP 172.31.23.23.8080 > 42.79.253.189.29729: Flags [.], ack 321, win 235, options [nop,nop,TS val 333436194 ecr 401990927], length 0

server 回應網頁內容 :

server 回應 : seq 從 1 開始並加上傳送 64 字元的長度, ack 為剛剛收到瀏覽器 header 的 seq:321

22:39:35.943037 IP 172.31.23.23.8080 > 42.79.253.189.29729: Flags [P.], seq 1:65, ack 321, win 235, options [nop,nop,TS val 333441694 ecr 401990927], length 64

server 像 browser 發送的 header 資料 :


server response header (64 bytes of data) :

HTTP/1.1 200 OK
Content-Length: 4
Content-Type: text/html

fghd

server 回應 : 資料傳送結束, 並等待結束回應

22:39:35.943086 IP 172.31.23.23.8080 > 42.79.253.189.29729: Flags [F.], seq 65, ack 321, win 235, options [nop,nop,TS val 333441694 ecr 401990927], length 0

瀏覽器發送 : 跟 server 說剛收到的 seq 為 65

22:39:36.067513 IP 42.79.253.189.29729 > 172.31.23.23.8080: Flags [.], ack 65, win 8201, options [nop,nop,TS val 402012885 ecr 333441694], length 0

瀏覽器發送 : 與上個封包一樣, 重複送了一次

22:39:36.067565 IP 42.79.253.189.29729 > 172.31.23.23.8080: Flags [.], ack 66, win 8201, options [nop,nop,TS val 402012885 ecr 333441694], length 0

瀏覽器向 server 發送結束連線

瀏覽器發送 : 回應剛剛 server 請求的結束, ack 66 為 seq 65 + 1, seq 321 為 ack 321

22:39:44.832134 IP 42.79.253.189.29729 > 172.31.23.23.8080: Flags [F.], seq 321, ack 66, win 8201, options [nop,nop,TS val 402021471 ecr 333441694], length 0

server 回應 : ack = seq + 1

22:39:44.832200 IP 172.31.23.23.8080 > 42.79.253.189.29729: Flags [.], ack 322, win 235, options [nop,nop,TS val 333443917 ecr 402021471], length 0

狀態代碼

  • CWR : CWR (Congestion Window Reduced) flag 為 1 代表接到 ECE 旗標 為 1 的 TCP 封包。
  • ECE : ECE (ECN-Echo) flag 為 1 代表 TCP peer (對等體) 具備 ECN (Explicit Congestion Notification, 明確擁塞通知) 功能,同時 IP 封包中的 ECN 欄位被設定為 11。
  • URG : URG (Urgent) flag 為 1 代表緊急封包,接收端應優先處理。
  • ACK : ACK (Acknowledgment) flag 為 1 代表此封包的 Acknowledgment Number 是有效的﹐是用來回應上一個封包。
  • PSH : PSH (Push function) flag 為 1 代表此封包連同緩衝區的其它相關封包應立即進行傳送,而無需等待緩衝區滿了才送。
  • RST : RST (Reset) flag 為 1 代表馬上結束連線,無需等待終止確認手續。
  • SYN : SYN (Synchronize) flag 為 1 代表要求雙方進行同步處理,也就是要求建立連線。
  • FIN : FIN (Finish) flag 為 1 代表資料傳送結束,等待結束回應,以便正式結束 TCP 傳送流程。

ref : http://120.105.184.250/peiyuli/network-3/%E7%B6%B2%E8%B7%AF%E7%B0%A1%E4%BB%8B%E8%88%87%E6%8C%87%E4%BB%A4.htm