Atom Feed

myersguo's blog 2018-04-19T01:19:22+00:00 myersguo Gin, Golang HTTP web framework 2018-04-18T00:00:00+00:00 /2018/04/18/golang-gin-http-web-framework <p>近期几个 api server 使用 <a href="https://github.com/gin-gonic/gin">gin</a> 框架来实现。这里记录下 gin 的基本使用。 <br /> 来看 gin 的<code class="highlighter-rouge">hello,world</code>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span> <span class="n">import</span> <span class="s2">"github.com/gin-gonic/gin"</span> <span class="n">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">r</span> <span class="p">:=</span> <span class="n">gin</span><span class="p">.</span><span class="n">Default</span><span class="p">()</span> <span class="n">r</span><span class="p">.</span><span class="n">GET</span><span class="p">(</span><span class="s2">"/ping"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span> <span class="s2">"message"</span><span class="p">:</span> <span class="s2">"pong"</span><span class="p">,</span> <span class="p">})</span> <span class="p">})</span> <span class="n">r</span><span class="p">.</span><span class="nf">Run</span><span class="p">()</span> <span class="p">//</span> <span class="n">listen</span> <span class="k">and</span> <span class="n">serve</span> <span class="n">on</span> <span class="m">0.0.0.0</span><span class="p">:</span><span class="m">8080</span> <span class="p">}</span> </code></pre></div></div> <p>ok, 如果我们使用原生的 <a href="https://golang.org/doc/articles/wiki/">http 服务</a>怎么写呢?</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span> <span class="n">import</span> <span class="p">(</span> <span class="s2">"net/http"</span> <span class="p">)</span> <span class="n">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">http</span><span class="p">.</span><span class="n">HandleFunc</span><span class="p">(</span><span class="s2">"/ping"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">w</span> <span class="n">http</span><span class="p">.</span><span class="n">ResponseWriter</span><span class="p">,</span> <span class="n">r</span> <span class="p">*</span><span class="n">http</span><span class="p">.</span><span class="n">Request</span><span class="p">)</span> <span class="p">{</span> <span class="n">w</span><span class="p">.</span><span class="n">Header</span><span class="p">().</span><span class="k">Set</span><span class="p">(</span><span class="s2">"Content-Type"</span><span class="p">,</span> <span class="s2">"application/json"</span><span class="p">)</span> <span class="n">w</span><span class="p">.</span><span class="n">Write</span><span class="p">([]</span><span class="n">byte</span><span class="p">(</span><span class="s2">"{</span><span class="se">\"</span><span class="s2">message</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">pong</span><span class="se">\"</span><span class="s2">}"</span><span class="p">))</span> <span class="p">})</span> <span class="n">http</span><span class="p">.</span><span class="n">ListenAndServe</span><span class="p">(</span><span class="s2">":9095"</span><span class="p">,</span> <span class="n">nil</span><span class="p">)</span> <span class="p">}</span> </code></pre></div></div> <p>看上去差不太多?再看 gin 提供的强大能力,把官方例子下载到本地: <br /> <code class="highlighter-rouge">curl https://raw.githubusercontent.com/gin-gonic/gin/master/examples/basic/main.go &gt; main.go</code></p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span> <span class="n">import</span> <span class="p">(</span> <span class="s2">"github.com/gin-gonic/gin"</span> <span class="p">)</span> <span class="n">var</span> <span class="n">DB</span> <span class="p">=</span> <span class="n">make</span><span class="p">(</span><span class="n">map</span><span class="p">[</span><span class="k">string</span><span class="p">]</span><span class="k">string</span><span class="p">)</span> <span class="n">func</span> <span class="n">setupRouter</span><span class="p">()</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Engine</span> <span class="p">{</span> <span class="p">//</span> <span class="n">Disable</span> <span class="n">Console</span> <span class="n">Color</span> <span class="p">//</span> <span class="n">gin</span><span class="p">.</span><span class="n">DisableConsoleColor</span><span class="p">()</span> <span class="n">r</span> <span class="p">:=</span> <span class="n">gin</span><span class="p">.</span><span class="n">Default</span><span class="p">()</span> <span class="p">//</span> <span class="n">Get</span> <span class="err">请求</span> <span class="n">r</span><span class="p">.</span><span class="n">GET</span><span class="p">(</span><span class="s2">"/ping"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="k">String</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="s2">"pong"</span><span class="p">)</span> <span class="p">//</span> <span class="err">返回字符串</span> <span class="p">})</span> <span class="p">//</span> <span class="n">Get</span> <span class="n">user</span> <span class="n">value</span> <span class="n">r</span><span class="p">.</span><span class="n">GET</span><span class="p">(</span><span class="s2">"/user/:name"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">user</span> <span class="p">:=</span> <span class="n">c</span><span class="p">.</span><span class="n">Params</span><span class="p">.</span><span class="n">ByName</span><span class="p">(</span><span class="s2">"name"</span><span class="p">)</span> <span class="p">//</span> <span class="err">获取参数</span> <span class="n">value</span><span class="p">,</span> <span class="n">ok</span> <span class="p">:=</span> <span class="n">DB</span><span class="p">[</span><span class="n">user</span><span class="p">]</span> <span class="k">if</span> <span class="n">ok</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span><span class="s2">"user"</span><span class="p">:</span> <span class="n">user</span><span class="p">,</span> <span class="s2">"value"</span><span class="p">:</span> <span class="n">value</span><span class="p">})</span> <span class="p">//</span> <span class="err">返回</span> <span class="n">JSON</span> <span class="err">数据</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span><span class="s2">"user"</span><span class="p">:</span> <span class="n">user</span><span class="p">,</span> <span class="s2">"status"</span><span class="p">:</span> <span class="s2">"no value"</span><span class="p">})</span> <span class="p">}</span> <span class="p">})</span> <span class="p">//</span> <span class="n">Authorized</span> <span class="n">group</span> <span class="p">(</span><span class="k">uses</span> <span class="n">gin</span><span class="p">.</span><span class="n">BasicAuth</span><span class="p">()</span> <span class="n">middleware</span><span class="p">)</span> <span class="p">//</span> <span class="n">Same</span> <span class="n">than</span><span class="p">:</span> <span class="p">//</span> <span class="n">authorized</span> <span class="p">:=</span> <span class="n">r</span><span class="p">.</span><span class="n">Group</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span> <span class="p">//</span> <span class="n">authorized</span><span class="p">.</span><span class="n">Use</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">BasicAuth</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">Credentials</span><span class="p">{</span> <span class="p">//</span> <span class="s2">"foo"</span><span class="p">:</span> <span class="s2">"bar"</span><span class="p">,</span> <span class="p">//</span> <span class="s2">"manu"</span><span class="p">:</span> <span class="s2">"123"</span><span class="p">,</span> <span class="p">//}))</span> <span class="n">authorized</span> <span class="p">:=</span> <span class="n">r</span><span class="p">.</span><span class="n">Group</span><span class="p">(</span><span class="s2">"/"</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">BasicAuth</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">Accounts</span><span class="p">{</span> <span class="s2">"foo"</span><span class="p">:</span> <span class="s2">"bar"</span><span class="p">,</span> <span class="p">//</span> <span class="n">user</span><span class="p">:</span><span class="n">foo</span> <span class="n">password</span><span class="p">:</span><span class="n">bar</span> <span class="s2">"manu"</span><span class="p">:</span> <span class="s2">"123"</span><span class="p">,</span> <span class="p">//</span> <span class="n">user</span><span class="p">:</span><span class="n">manu</span> <span class="n">password</span><span class="p">:</span><span class="m">123</span> <span class="p">}))</span> <span class="p">//</span> <span class="err">认证检查</span> <span class="n">authorized</span><span class="p">.</span><span class="n">POST</span><span class="p">(</span><span class="s2">"admin"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">user</span> <span class="p">:=</span> <span class="n">c</span><span class="p">.</span><span class="n">MustGet</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">AuthUserKey</span><span class="p">).(</span><span class="k">string</span><span class="p">)</span> <span class="p">//</span> <span class="n">Parse</span> <span class="n">JSON</span> <span class="n">var</span> <span class="n">json</span> <span class="n">struct</span> <span class="p">{</span> <span class="n">Value</span> <span class="k">string</span> <span class="p">`</span><span class="n">json</span><span class="p">:</span><span class="s2">"value"</span> <span class="n">binding</span><span class="p">:</span><span class="s2">"required"</span><span class="p">`</span> <span class="p">}</span> <span class="k">if</span> <span class="n">c</span><span class="p">.</span><span class="n">Bind</span><span class="p">(&amp;</span><span class="n">json</span><span class="p">)</span> <span class="p">==</span> <span class="n">nil</span> <span class="p">{</span> <span class="n">DB</span><span class="p">[</span><span class="n">user</span><span class="p">]</span> <span class="p">=</span> <span class="n">json</span><span class="p">.</span><span class="n">Value</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span><span class="s2">"status"</span><span class="p">:</span> <span class="s2">"ok"</span><span class="p">})</span> <span class="p">}</span> <span class="p">})</span> <span class="n">return</span> <span class="n">r</span> <span class="p">}</span> <span class="n">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">r</span> <span class="p">:=</span> <span class="n">setupRouter</span><span class="p">()</span> <span class="p">//</span> <span class="n">Listen</span> <span class="k">and</span> <span class="n">Server</span> <span class="k">in</span> <span class="m">0.0.0.0</span><span class="p">:</span><span class="m">8080</span> <span class="n">r</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="s2">":9095"</span><span class="p">)</span> <span class="p">}</span> </code></pre></div></div> <h3 id="初探">初探</h3> <p>看上面的例子,你几乎只需要定义好 path, handlefunc 即可:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>type HandlerFunc func(*Context) </code></pre></div></div> <p>gin 实现了 servehttp 的方法:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// Conforms to the http.Handler interface. func (engine *Engine) ServeHTTP(w http.ResponseWriter, req *http.Request) { c := engine.pool.Get().(*Context) c.writermem.reset(w) c.Request = req c.reset() engine.handleHTTPRequest(c) engine.pool.Put(c) } </code></pre></div></div> <p>详细的 path -&gt; handle 的逻辑这里不详细展开.</p> <h3 id="参数获取">参数获取</h3> <p>获取 path 中变量:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>router.GET("/user/:name", func(c *gin.Context) { name := c.Param("name") c.String(http.StatusOK, "Hello %s", name) }) </code></pre></div></div> <p>获取URL中参数:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> // The request responds to a url matching: /welcome?firstname=Jane&amp;lastname=Doe router.GET("/welcome", func(c *gin.Context) { firstname := c.DefaultQuery("firstname", "Guest") lastname := c.Query("lastname") // shortcut for c.Request.URL.Query().Get("lastname") c.String(http.StatusOK, "Hello %s %s", firstname, lastname) }) </code></pre></div></div> <p>获取 POST 请求参数:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>//POST /post?id=1234&amp;page=1 HTTP/1.1 //Content-Type: application/x-www-form-urlencoded // //name=manu&amp;message=this_is_great router.POST("/form_post", func(c *gin.Context) { message := c.PostForm("message") nick := c.DefaultPostForm("nick", "anonymous") c.JSON(200, gin.H{ "status": "posted", "message": message, "nick": nick, }) }) </code></pre></div></div> <p>获取上传文件:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file, _ := c.FormFile("file") //or form, _ := c.MultipartForm() files := form.File["upload[]"] </code></pre></div></div> <h3 id="返回">返回</h3> <p>HTML :</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>router.LoadHTMLGlob("templates/*") c.HTML(http.StatusOK, "index.tmpl", gin.H{ "title": "Main website", }) </code></pre></div></div> <p>JSON 格式:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>c.JSON(http.StatusOK, gin.H{"message": "hey", "status": http.StatusOK}) var msg struct { Name string `json:"user"` Message string Number int } msg.Name = "Lena" msg.Message = "hey" msg.Number = 123 // Note that msg.Name becomes "user" in the JSON // Will output : {"user": "Lena", "Message": "hey", "Number": 123} c.JSON(http.StatusOK, msg) </code></pre></div></div> <p>XML:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>c.XML(http.StatusOK, gin.H{"message": "hey", "status": http.StatusOK}) </code></pre></div></div> <p>静态文件:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>router.Static("/assets", "./assets") </code></pre></div></div> <h3 id="状态码">状态码</h3> <p>重定向:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>c.Redirect(http.StatusMovedPermanently, "http://www.google.com/") </code></pre></div></div> <h3 id="使用中间件">使用中间件</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>func (engine *Engine) Use(middleware ...HandlerFunc) IRoutes { engine.RouterGroup.Use(middleware...) engine.rebuild404Handlers() engine.rebuild405Handlers() return engine } </code></pre></div></div> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>r.Use(gin.Recovery()) </code></pre></div></div> <h3 id="注意">注意</h3> <blockquote> <p>When starting new Goroutines inside a middleware or handler, you SHOULD NOT use the original context inside it, you have to use a read-only copy.</p> </blockquote> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>r.GET("/long_async", func(c *gin.Context) { // create copy to be used inside the goroutine cCp := c.Copy() go func() { // simulate a long task with time.Sleep(). 5 seconds time.Sleep(5 * time.Second) // note that you are using the copied context "cCp", IMPORTANT log.Println("Done! in path " + cCp.Request.URL.Path) }() }) </code></pre></div></div> <h3 id="多服务">多服务</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span> <span class="n">import</span> <span class="p">(</span> <span class="s2">"log"</span> <span class="s2">"net/http"</span> <span class="s2">"time"</span> <span class="s2">"github.com/gin-gonic/gin"</span> <span class="s2">"golang.org/x/sync/errgroup"</span> <span class="p">)</span> <span class="n">var</span> <span class="p">(</span> <span class="n">g</span> <span class="n">errgroup</span><span class="p">.</span><span class="n">Group</span> <span class="p">)</span> <span class="n">func</span> <span class="n">router01</span><span class="p">()</span> <span class="n">http</span><span class="p">.</span><span class="n">Handler</span> <span class="p">{</span> <span class="n">e</span> <span class="p">:=</span> <span class="n">gin</span><span class="p">.</span><span class="n">New</span><span class="p">()</span> <span class="n">e</span><span class="p">.</span><span class="n">Use</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">Recovery</span><span class="p">())</span> <span class="n">e</span><span class="p">.</span><span class="n">GET</span><span class="p">(</span><span class="s2">"/"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span> <span class="n">http</span><span class="p">.</span><span class="n">StatusOK</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span> <span class="s2">"code"</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">StatusOK</span><span class="p">,</span> <span class="s2">"error"</span><span class="p">:</span> <span class="s2">"Welcome server 01"</span><span class="p">,</span> <span class="p">},</span> <span class="p">)</span> <span class="p">})</span> <span class="n">return</span> <span class="n">e</span> <span class="p">}</span> <span class="n">func</span> <span class="n">router02</span><span class="p">()</span> <span class="n">http</span><span class="p">.</span><span class="n">Handler</span> <span class="p">{</span> <span class="n">e</span> <span class="p">:=</span> <span class="n">gin</span><span class="p">.</span><span class="n">New</span><span class="p">()</span> <span class="n">e</span><span class="p">.</span><span class="n">Use</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">Recovery</span><span class="p">())</span> <span class="n">e</span><span class="p">.</span><span class="n">GET</span><span class="p">(</span><span class="s2">"/"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span> <span class="n">http</span><span class="p">.</span><span class="n">StatusOK</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span> <span class="s2">"code"</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">StatusOK</span><span class="p">,</span> <span class="s2">"error"</span><span class="p">:</span> <span class="s2">"Welcome server 02"</span><span class="p">,</span> <span class="p">},</span> <span class="p">)</span> <span class="p">})</span> <span class="n">return</span> <span class="n">e</span> <span class="p">}</span> <span class="n">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">server01</span> <span class="p">:=</span> <span class="p">&amp;</span><span class="n">http</span><span class="p">.</span><span class="n">Server</span><span class="p">{</span> <span class="n">Addr</span><span class="p">:</span> <span class="s2">":8080"</span><span class="p">,</span> <span class="n">Handler</span><span class="p">:</span> <span class="n">router01</span><span class="p">(),</span> <span class="n">ReadTimeout</span><span class="p">:</span> <span class="m">5</span> <span class="p">*</span> <span class="n">time</span><span class="p">.</span><span class="n">Second</span><span class="p">,</span> <span class="n">WriteTimeout</span><span class="p">:</span> <span class="m">10</span> <span class="p">*</span> <span class="n">time</span><span class="p">.</span><span class="n">Second</span><span class="p">,</span> <span class="p">}</span> <span class="n">server02</span> <span class="p">:=</span> <span class="p">&amp;</span><span class="n">http</span><span class="p">.</span><span class="n">Server</span><span class="p">{</span> <span class="n">Addr</span><span class="p">:</span> <span class="s2">":8081"</span><span class="p">,</span> <span class="n">Handler</span><span class="p">:</span> <span class="n">router02</span><span class="p">(),</span> <span class="n">ReadTimeout</span><span class="p">:</span> <span class="m">5</span> <span class="p">*</span> <span class="n">time</span><span class="p">.</span><span class="n">Second</span><span class="p">,</span> <span class="n">WriteTimeout</span><span class="p">:</span> <span class="m">10</span> <span class="p">*</span> <span class="n">time</span><span class="p">.</span><span class="n">Second</span><span class="p">,</span> <span class="p">}</span> <span class="n">g</span><span class="p">.</span><span class="n">Go</span><span class="p">(</span><span class="n">func</span><span class="p">()</span> <span class="n">error</span> <span class="p">{</span> <span class="n">return</span> <span class="n">server01</span><span class="p">.</span><span class="n">ListenAndServe</span><span class="p">()</span> <span class="p">})</span> <span class="n">g</span><span class="p">.</span><span class="n">Go</span><span class="p">(</span><span class="n">func</span><span class="p">()</span> <span class="n">error</span> <span class="p">{</span> <span class="n">return</span> <span class="n">server02</span><span class="p">.</span><span class="n">ListenAndServe</span><span class="p">()</span> <span class="p">})</span> <span class="k">if</span> <span class="n">err</span> <span class="p">:=</span> <span class="n">g</span><span class="p">.</span><span class="nf">Wait</span><span class="p">();</span> <span class="n">err</span> <span class="c1">!= nil { </span> <span class="nb">log</span><span class="p">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <p>未完待续</p> Ali Cloud Set Up 2018-04-03T00:00:00+00:00 /2018/04/03/alicloud-setup <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>useradd -m -d /home/myersguo -s /bin/bash myersguo passwd myersguo sudo usermod -a -G sudo myersguo sed -i "s/PasswordAuthentication yes/PasswordAuthentication no/g" /etc/ssh/sshd_config mkdir /home/myersguo/data /home/myersguo/app /home/myersguo/logs /home/myersguo/logs/applogs /home/myersguo/logs/php sudo apt-get update apt-get install net-tools apt-get install libpcre3-dev libssl-dev perl make build-essential curl </code></pre></div></div> <p>镜像地址:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># /etc/apt/sources.list deb http://mirrors.cloud.aliyuncs.com/debian/ stretch main contrib non-free deb http://mirrors.cloud.aliyuncs.com/debian/ stretch-updates main contrib non-free deb http://mirrors.cloud.aliyuncs.com/debian/ stretch-proposed-updates main non-free contrib deb http://mirrors.cloud.aliyuncs.com/debian/ stretch-backports main non-free contrib ## Major bug fix updates produced after the final release of the ## distribution. deb-src http://mirrors.cloud.aliyuncs.com/debian/ stretch-updates main contrib non-free deb-src http://mirrors.cloud.aliyuncs.com/debian/ stretch main contrib non-free deb-src http://mirrors.cloud.aliyuncs.com/debian/ stretch-proposed-updates main contrib non-free deb-src http://mirrors.cloud.aliyuncs.com/debian/ stretch-backports main contrib non-free deb http://mirrors.cloud.aliyuncs.com/debian-security/ stretch/updates main non-free contrib deb-src http://mirrors.cloud.aliyuncs.com/debian-security/ stretch/updates main non-free contrib </code></pre></div></div> <h3 id="mysql">mysql</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get install -y mysql-server mysql-client mysql_secure_installation /etc/mysql/my.cnf /etc/systemd/system/mysql.service /etc/systemd/system/mysqld.service /etc/systemd/system/multi-user.target.wants/mariadb.service </code></pre></div></div> <h3 id="git">git</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt-get install git-core </code></pre></div></div> <h3 id="nginxopenresty">nginx/openresty</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get install libpcre3-dev \ libssl-dev perl make build-essential curl PATH=/usr/local/openresty/nginx/sbin:$PATH apt-get install -y nginx-full lsof -i :80 </code></pre></div></div> <h3 id="setup-my-blog">setup my blog</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt-get install ruby-full git clone https://github.com/myersguo/myersguo.github.com.git </code></pre></div></div> <p>参考<a href="https://help.github.com/articles/setting-up-your-github-pages-site-locally-with-jekyll/">例子</a>, 生成静态文件.启动 nginx 配置:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>error_log logs/error.log; events { worker_connections 1024; } http { server { listen 80; root /home/myersguo/app/myersguo.github.com/_site; include /usr/local/openresty/nginx/conf/mime.types; default_type application/octet-stream; index index.html index.htm; location / { } } } </code></pre></div></div> <p>启动: <code class="highlighter-rouge">nginx -p /home/myersguo -c conf/nginx.conf</code></p> About HyperLogLog 2018-03-12T00:00:00+00:00 /2018/03/12/about-HyperLogLog <h3 id="统计">统计</h3> <p>我们之前谈到过<a href="https://myersguo.github.io/2017/10/15/redis-bitmap-2.html">日活统计</a>的问题:</p> <p>假如直接用集合统计<code class="highlighter-rouge">sadd uv user_1</code>,那么要存储 1亿数据的用户数据,占用内存将达到<strong>1.02G</strong>(uid 按照11位存储,11<em>8</em>100000000)。如果采用 bitmap 的形式 <code class="highlighter-rouge">SETBIT uv 1 1</code>,那么存储1亿数据,内存将需要12M( 100000000 bit), 大大节省了空间。但是,如果 user_id 不稀疏,11 位的uid 将占用1.16G (10000000000) 空间。从之前的文章,我们知道在大数统计上,使用<strong>HyperLogLog</strong>算法来统计占用空间将大大缩短。我们来解开它的面纱。</p> <h3 id="基数估算">基数估算</h3> <p>上面我们提到,11 位的 uid 的 bit 存储大概占用1.16G,能表示 100亿的数据,但是我们通过的用户数都在10亿以内,我们仅仅使用10位bit就可以表示。那我们的为题就变成了,如果用10位bit来表示最大100亿的10以内的数据。</p> <p>显然,如果有一个 hash 函数,将100亿的数据,均匀的 hash 到10位 bit 内即可。</p> <p>1990年,一篇「A linear-time probabilistic counting algorithm for database applications」提出这种线性估算法。</p> <blockquote> <p>设有一哈希函数H,其哈希结果空间有m个值(最小值0,最大值m-1),并且哈希结果服从均匀分布。使用一个长度为m的bitmap,每个bit为一个桶,均初始化为0,设一个集合的基数为n,此集合所有元素通过H哈希到bitmap中,如果某一个元素被哈希到第k个比特并且第k个比特为0,则将其置为1。<a href="http://blog.codinglabs.org/articles/algorithms-for-cardinality-estimation-part-ii.html">link</a></p> </blockquote> <p><a href="https://gist.github.com/devdazed/3873524">这里</a> or <a href="https://github.com/addthis/stream-lib/blob/master/src/main/java/com/clearspring/analytics/stream/cardinality/LinearCounting.java">这里</a>有实现的例子。</p> <h3 id="hyperloglog">HyperLogLog</h3> <p>这篇<a href="http://content.research.neustar.biz/blog/hll.html">blog</a>演示了 hyperloglog 算法的过程:</p> <p>初始化一个长度为m(64)的数组,将待统计的集合分为m组, 每组的统计值放到对应的数组位置上。</p> <p>输入 value, 通过 hash 得到 hash value;<br /> hash value 的最后24位用二进制表示;<br /> 二进制bit位最后r位bit位,表示落在数组(桶)的位置index。<br /> 从第r+1位开始,第一个1出现的值,放到第index个位置。</p> <blockquote> <p>分桶平均的基本原理是将统计数据划分为mm个桶,每个桶分别统计各自的{k_{max}}k</p> </blockquote> <blockquote> <p>就是将哈希空间平均分成m份,每份称之为一个桶(bucket)。对于每一个元素,其哈希值的前k比特作为桶编号,其中2^k=m,而后L-k个比特作为真正用于基数估计的比特串。桶编号相同的元素被分配到同一个桶,在进行基数估计时,首先计算每个桶内元素最大的第一个“1”的位置,设为M[i],然后对这m个值取平均后再进行估计,即:</p> <p>n̂ =2^(1/m∑M[i]) 这相当于物理试验中经常使用的多次试验取平均的做法,可以有效消减因偶然性带来的误差。</p> </blockquote> <h3 id="参考资料">参考资料</h3> <p><a href="http://blog.codinglabs.org/articles/algorithms-for-cardinality-estimation-part-iii.html">http://blog.codinglabs.org/articles/algorithms-for-cardinality-estimation-part-iii.html</a> <br /> <a href="https://github.com/flier/pyfasthash">pyhasher</a><br /> <a href="http://www.rainybowe.com/blog/2017/07/13/%E7%A5%9E%E5%A5%87%E7%9A%84HyperLogLog%E7%AE%97%E6%B3%95/index.html">http://www.rainybowe.com/blog/2017/07/13/%E7%A5%9E%E5%A5%87%E7%9A%84HyperLogLog%E7%AE%97%E6%B3%95/index.html</a></p> python time and date 2018-03-09T00:00:00+00:00 /2018/03/09/python-time-date <h3 id="convert-str-to-timestamp">convert str to timestamp</h3> <p>如果在 PHP 中,直接使用:</p> <p><code class="highlighter-rouge">strtotime("2018-01-01")</code></p> <p>但在<code class="highlighter-rouge">python</code>中,这似乎成了一件头疼的事情。</p> <p>python 的<code class="highlighter-rouge">time</code> 是什么的?</p> <p><code class="highlighter-rouge">time</code> 这个 module 提供了函数管理时间值。有两种方式表示:</p> <ul> <li>seconds since the Epoch,in UTC</li> <li>tuple of 9 integers (year, month, day, hours, minutes, seconds, weekday, Julian day, DST flag)</li> </ul> <p>我们能用它来做什么呢?</p> <p>time.time() — 返回当前时间戳,可恨的是,它不支持参数,只能返回当前的,如果想要返回其他时间段的不可以</p> <p>time.localtime() — 返回当前的时间time.struct_time类型</p> <p>两种表示怎么转换呢?</p> <h4 id="str-to-timestamp">str to timestamp</h4> <p><strong>strptime-&gt;time.mktime</strong></p> <p>time.mktime(time.localtime())</p> <p>你如果需要将’2018-01-01’转换成时间戳,需要先用 tuple 表示:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>time.strptime("2018-01-01", "%Y-%m-%d") </code></pre></div></div> <p>然后转换成时间:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>time.mktime(time.strptime("2018-01-01")) </code></pre></div></div> <h3 id="timestamp-to-str">timestamp to str</h3> <p>先将时间戳转化成struct time:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>time.localtime(1520578947.458189) </code></pre></div></div> <p>再转成 string <strong>time.localtime-&gt;time.strftime</strong>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># 格式化 time.strftime("%Y-%m-%d %H:%M:%S +0000", time.localtime(1520578947.458189)) # 直接返回 string time.asctime(time.localtime(1520578947.45889)) </code></pre></div></div> <p>可以看到,没有一个直接的 str2time 之类的方法,而是要有一个中间的 <code class="highlighter-rouge">struct_time</code> 阶段,WHY?</p> <h3 id="datetime">datetime</h3> <ul> <li>date(year, month, day) –&gt; date object</li> <li>datetime(year, month, day[, hour[, minute[, second[, microsecond[,tzinfo]]]]])</li> <li>tzinfo: bstract base class for time zone info objects</li> <li>time([hour[, minute[, second[, microsecond[, tzinfo]]]]]) –&gt; a time object</li> <li>timedelta: Difference between two datetime values</li> </ul> <h4 id="时区转换">时区转换</h4> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from dateutil import tz src = datetime.datetime(2018,03,11) from_zone = tz.gettz('UTC') src_inzone = src.replace(tzinfo=from_zone) to_zone = tz.gettz('CST') src_inzone.astimezone(tz=to_zone) </code></pre></div></div> <h4 id="timestamp--string-datetime">timestamp string datetime</h4> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># datetime to string a = datetime.datetime(2018,01,01) a.strftime('%Y-%m-%d') # string to datetime datetime.datetime.strptime('20180101','%Y%m%d') # timestamp to datetime datetime.datetime.fromtimestamp(1520746754) # datetime to timestamp a.strftime('%s') # datetime to time tuple a.timetuple() # current datetime datetime.datetime.today(), datetime.datetime.now() </code></pre></div></div> 你的灯亮着吗?读书笔记 2018-03-06T00:00:00+00:00 /2018/03/06/are-your-lights-on-reading <p><a href="https://book.douban.com/subject/25772550/">这本书</a>将了我们应该怎么去定义问题,看清问题的方法:</p> <ul> <li>问题是什么?</li> <li>问题该由谁解决?</li> <li>问题来自哪里?</li> <li>问题真的需要解决吗?</li> </ul> <h3 id="问题的定义">问题的定义?</h3> <p><strong>每一个解决方案都是下一个问题的来源。</strong>需求要持续迭代,程序要不断的优化。</p> <p><strong>某些问题最困难的部分在于发现问题的所在。</strong> 程序设计中,潜在的风险,或者上线后的迭代会不会暴露当前设计的缺陷(可扩展性)?不同的角色的<strong>痛点</strong>在同一解决方案下可能截然相反。</p> <p><strong>没有完美的解决方案,如果想不出至少3个可能出错的地方,你就没有真正的理解这个问题</strong>。比如最近的一个需求迭代:把放量单位从万更改为个,如果你想到了更改的方法,那针对你的方案思考可能出现的问题:1)当前的放量超量 2)当前的放量失效 3)用户的操作前后不一致。等等。这写问题转换的「成本」问题应该同时被解决。</p> About sqlalchemy 2018-02-22T00:00:00+00:00 /2018/02/22/about-sqlalchemy <p>从<a href="http://docs.sqlalchemy.org/en/latest/orm/tutorial.html">这个 tutorial</a>,我们来看一下使用 sqlalchemy 的基本用法。</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from sqlalchemy import create_engine from sqlalchemy.orm import sessionmaker from sqlalchemy.ext.declarative import declarative_base from sqlalchemy import Column, Integer, String Base = declarative_base() # Declare a Mapping class User(Base): __tablename__ = 'users' id = Column(Integer, primary_key=True) name = Column(String(50)) fullname = Column(String(50)) password = Column(String(12)) def __repr__(self): return "&lt;User(name='%s', fullname='%s', password='%s')&gt;" % ( self.name, self.fullname, self.password) class UserScore(Base): __tablename__ = 'user_scores' id = Column(Integer, primary_key=True) user_id = Column(Integer) score = Column(Integer) # Connecting engine engine = create_engine('sqlite:///:memory:', echo=True) # Creating a Session Session = sessionmaker(bind=engine) session = Session() # Create a Schema(table) Base.metadata.create_all(engine) # Create an Instance of the Mapped Class ed_user = User(name='ed', fullname='Ed Jones', password='edspassword') # Adding and Updating Objects session.add(ed_user) our_user = session.query(User).filter_by(name='ed').first() print our_user ed_user.password = 'f8s7ccs' session.commit() # Score score = UserScore(user_id=ed_user.id, score=12) session.add(score) session.commit() our_user = session.query(User).filter_by(name='ed').first() print our_user # Sub Query stmt = session.query(UserScore.user_id).group_by(UserScore.user_id).subquery() print session.query(User, stmt.c.user_id).join(stmt, User.id == stmt.c.user_id).all() """ sqlalchemy.engine.base.Engine SELECT users.id AS users_id, users.name AS users_name, users.fullname AS users_fullname, users.password AS users_password, anon_1.user_id AS anon_1_user_id FROM users JOIN (SELECT user_scores.user_id AS user_id FROM user_scores GROUP BY user_scores.user_id) AS anon_1 ON users.id = anon_1.user_id """ </code></pre></div></div> <p>我看来看下它的整体架构:</p> <p><a href="http://docs.sqlalchemy.org/en/latest/core/engines.html"><img src="http://docs.sqlalchemy.org/en/latest/_images/sqla_engine_arch.png" alt="SQLAlchemy 架构" /></a></p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── __init__.py ├── connectors │   ├── __init__.py │   ├── mxodbc.py │   ├── pyodbc.py │   └── zxJDBC.py ├── cprocessors.so ├── cresultproxy.so ├── cutils.so ├── databases │   └── __init__.py ├── dialects │   ├── __init__.py │   ├── firebird │   │   ├── __init__.py │   │   ├── base.py │   │   ├── fdb.py │   │   └── kinterbasdb.py │   ├── mssql │   │   ├── __init__.py │   │   ├── adodbapi.py │   │   ├── base.py │   │   ├── information_schema.py │   │   ├── mxodbc.py │   │   ├── pymssql.py │   │   ├── pyodbc.py │   │   └── zxjdbc.py │   ├── mysql │   │   ├── __init__.py │   │   ├── base.py │   │   ├── cymysql.py │   │   ├── dml.py │   │   ├── enumerated.py │   │   ├── gaerdbms.py │   │   ├── json.py │   │   ├── mysqlconnector.py │   │   ├── mysqldb.py │   │   ├── oursql.py │   │   ├── pymysql.py │   │   ├── pyodbc.py │   │   ├── reflection.py │   │   ├── types.py │   │   └── zxjdbc.py │   ├── oracle │   │   ├── __init__.py │   │   ├── base.py │   │   ├── cx_oracle.py │   │   └── zxjdbc.py │   ├── postgresql │   │   ├── __init__.py │   │   ├── array.py │   │   ├── base.py │   │   ├── dml.py │   │   ├── ext.py │   │   ├── hstore.py │   │   ├── json.py │   │   ├── pg8000.py │   │   ├── psycopg2.py │   │   ├── psycopg2cffi.py │   │   ├── pygresql.py │   │   ├── pypostgresql.py │   │   ├── ranges.py │   │   └── zxjdbc.py │   ├── sqlite │   │   ├── __init__.py │   │   ├── base.py │   │   ├── pysqlcipher.py │   │   └── pysqlite.py │   └── sybase │   ├── __init__.py │   ├── base.py │   ├── mxodbc.py │   ├── pyodbc.py │   └── pysybase.py ├── engine │   ├── __init__.py │   ├── base.py (engine &amp; connection, Provides high-level functionality for a wrapped DB-API connection) │   ├── default.py │   ├── interfaces.py │   ├── reflection.py │   ├── result.py │   ├── strategies.py │   ├── threadlocal.py │   ├── url.py │   └── util.py ├── event │   ├── __init__.py │   ├── api.py │   ├── attr.py │   ├── base.py │   ├── legacy.py │   └── registry.py ├── events.py ├── exc.py ├── ext │   ├── __init__.py │   ├── associationproxy.py │   ├── automap.py │   ├── baked.py │   ├── compiler.py │   ├── declarative │   │   ├── __init__.py │   │   ├── api.py │   │   ├── base.py │   │   └── clsregistry.py │   ├── horizontal_shard.py │   ├── hybrid.py │   ├── indexable.py │   ├── instrumentation.py │   ├── mutable.py │   ├── orderinglist.py │   └── serializer.py ├── inspection.py ├── interfaces.py ├── log.py ├── orm │   ├── __init__.py │   ├── attributes.py │   ├── base.py │   ├── collections.py │   ├── dependency.py │   ├── deprecated_interfaces.py │   ├── descriptor_props.py │   ├── dynamic.py │   ├── evaluator.py │   ├── events.py │   ├── exc.py │   ├── identity.py │   ├── instrumentation.py │   ├── interfaces.py │   ├── loading.py (convert database rows into object instances and associated state) │   ├── mapper.py │   ├── path_registry.py │   ├── persistence.py │   ├── properties.py │   ├── query.py (central construct useed by the orm to construct database queries,查询类) │   ├── relationships.py │   ├── scoping.py │   ├── session.py (Manages persistence operations for ORM-mapped objects) │   ├── state.py │   ├── strategies.py │   ├── strategy_options.py │   ├── sync.py │   ├── unitofwork.py │   └── util.py ├── pool.py ├── processors.py ├── schema.py ( describes a database entity,provides the building blocks for database metadata) ├── sql (sql表达式) │   ├── __init__.py │   ├── annotation.py │   ├── base.py │   ├── compiler.py │   ├── crud.py │   ├── ddl.py │   ├── default_comparator.py │   ├── dml.py │   ├── elements.py │   ├── expression.py │   ├── functions.py │   ├── naming.py │   ├── operators.py │   ├── schema.py │   ├── selectable.py │   ├── sqltypes.py │   ├── type_api.py │   ├── util.py │   └── visitors.py ├── testing │   ├── __init__.py │   ├── assertions.py │   ├── assertsql.py │   ├── config.py │   ├── engines.py │   ├── entities.py │   ├── exclusions.py │   ├── fixtures.py │   ├── mock.py │   ├── pickleable.py │   ├── plugin │   │   ├── __init__.py │   │   ├── bootstrap.py │   │   ├── noseplugin.py │   │   ├── plugin_base.py │   │   └── pytestplugin.py │   ├── profiling.py │   ├── provision.py │   ├── replay_fixture.py │   ├── requirements.py │   ├── runner.py │   ├── schema.py │   ├── suite │   │   ├── __init__.py │   │   ├── test_ddl.py │   │   ├── test_dialect.py │   │   ├── test_insert.py │   │   ├── test_reflection.py │   │   ├── test_results.py │   │   ├── test_select.py │   │   ├── test_sequence.py │   │   ├── test_types.py │   │   └── test_update_delete.py │   ├── util.py │   └── warnings.py ├── types.py(defines genericized SQL types) └── util ├── __init__.py ├── _collections.py ├── compat.py ├── deprecations.py ├── langhelpers.py ├── queue.py └── topological.py </code></pre></div></div> <p>看一下<code class="highlighter-rouge">query</code>的最终的返回部分:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def __iter__(self): context = self._compile_context() context.statement.use_labels = True if self._autoflush and not self._populate_existing: self.session._autoflush() return self._execute_and_instances(context) def _execute_and_instances(self, querycontext): conn = self._connection_from_session( mapper=self._mapper_zero_or_none(), clause=querycontext.statement, close_with_result=True) result = conn.execute(querycontext.statement, self._params) return loading.instances(self, result, querycontext) </code></pre></div></div> <h3 id="session-connection">session connection</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def connection(self, mapper=None, clause=None, bind=None, close_with_result=False, **kw): if bind is None: bind = self.get_bind(mapper, clause=clause, **kw) return self._connection_for_bind(bind, close_with_result=close_with_result) def _connection_for_bind(self, engine, **kwargs): if self.transaction is not None: return self.transaction._connection_for_bind(engine) else: return engine.contextual_connect(**kwargs) # connection, engine.default def connect(self, *cargs, **cparams): return self.dbapi.connect(*cargs, **cparams) </code></pre></div></div> <h3 id="pool">pool</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> pool = kwargs.pop('pool', None) if pool is None: def connect(): try: return dialect.connect(*cargs, **cparams) except dialect.dbapi.Error, e: invalidated = dialect.is_disconnect(e, None, None) # Py3K #raise exc.DBAPIError.instance(None, None, # e, dialect.dbapi.Error, # connection_invalidated=invalidated #) from e # Py2K import sys raise exc.DBAPIError.instance( None, None, e, dialect.dbapi.Error, connection_invalidated=invalidated ), None, sys.exc_info()[2] # end Py2K creator = kwargs.pop('creator', connect) poolclass = kwargs.pop('poolclass', None) if poolclass is None: poolclass = dialect_cls.get_pool_class(u) </code></pre></div></div> python http 请求编码问题追查 2018-02-06T00:00:00+00:00 /2018/02/06/python-unicode-error <p>背景: <br /> 有一个上传文件的业务,偶尔会报错: <br /> <code class="highlighter-rouge">'utf8' codec can't decode byte 0xce in position 17: invalid continuation byte</code></p> <p>问题追查,我们看下堆栈上下文:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import traceback traceback.print_exc() </code></pre></div></div> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> File "/usr/lib/python2.7/httplib.py", line 1039, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1073, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 1035, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 877, in _send_output msg += message_body UnicodeDecodeError: 'utf8' codec can't decode byte 0xce in position 17: invalid continuation byte </code></pre></div></div> <p>ok,我们看到这里是因为字符串相加造成的。看到这里就是经典的 python unicode 和 string 的问题了。我们先补充下基础知识,先看一个例子:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>["{:02x}".format(ord(c)) for c in '我'] ['e6', '88', '91'] ["{:02x}".format(ord(c)) for c in u'我'] ['6211'] [ord(c) for c in u'我'] [25105] </code></pre></div></div> <p>因为计算机无法表示’我’,只能用编码方式表示。所以系统需要一个默认的编码方式,可通过:<code class="highlighter-rouge">sys.setdefaultencoding('UTF8')</code>来设置。ok, 那:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>'我' + u'我' </code></pre></div></div> <p>就相当于:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>'我'.decode('ascii') + u'我' </code></pre></div></div> <p>当然会报错。</p> <p>回到我们最初的问题,我们来看下<code class="highlighter-rouge">httplib</code>的代码:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def request(self, method, url, body=None, headers={}): """Send a complete request to the server.""" self._send_request(method, url, body, headers) def _send_request(self, method, url, body, headers): # Honor explicitly requested Host: and Accept-Encoding: headers. header_names = dict.fromkeys([k.lower() for k in headers]) skips = {} if 'host' in header_names: skips['skip_host'] = 1 if 'accept-encoding' in header_names: skips['skip_accept_encoding'] = 1 self.putrequest(method, url, **skips) if body is not None and 'content-length' not in header_names: self._set_content_length(body) for hdr, value in headers.iteritems(): self.putheader(hdr, value) self.endheaders(body) def _send_output(self, message_body=None): """Send the currently buffered request and clear the buffer. Appends an extra \\r\\n to the buffer. A message_body may be specified, to be appended to the request. """ self._buffer.extend(("", "")) msg = "\r\n".join(self._buffer) del self._buffer[:] # If msg and message_body are sent in a single send() call, # it will avoid performance problems caused by the interaction # between delayed ack and the Nagle algorithm. if isinstance(message_body, str): msg += message_body message_body = None self.send(msg) if message_body is not None: #message_body was not a string (i.e. it is a file) and #we must run the risk of Nagle self.send(message_body) </code></pre></div></div> <p>可以看到: <code class="highlighter-rouge">msg += message_body</code>,其中<code class="highlighter-rouge">msg</code>是http的请求头的相关信息,而message_body 是请求体.</p> <p>而这里必须是 ascii 编码。</p> <p>我们写个 demo 复现一下问题:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># coding: utf8 import sys reload(sys) sys.setdefaultencoding('UTF8') from httplib import HTTPConnection conn = HTTPConnection('www.baidu.com') conn.request('GET', '/s=xxxx', '我\xce\xe6\x88\x91') print conn.getresponse().read() conn = HTTPConnection('www.baidu.com') conn.request('GET', u'/s=xxxx', '我\xce\xe6\x88\x91') print conn.getresponse().read() </code></pre></div></div> <p>ok,上面的输出没有问题,下面的输出,就会直接报错。因为我们这里的url 会拼接到请求头中,没有使用 ascii。</p> <p>(done)</p> Hello,React 2018-01-29T00:00:00+00:00 /2018/01/29/hello-react <h3 id="认知">认知</h3> <p>从<code class="highlighter-rouge">jquery</code>到<code class="highlighter-rouge">angularjs</code>,再切换到<code class="highlighter-rouge">react</code>你能感到一种「工程方式」的转变。在 react 全家桶下,写的代码更像是照着「模具」画像(工程感更强烈)。React 的一些新认知:</p> <p><code class="highlighter-rouge">组件和DOM树</code>: 浏览器把CSS和HTML解析成DOM树,而React同样把HTML结构解析成了DOM数,不过更加轻量级。DOM树的每一个节点变化由React对象管理。 <br /> <code class="highlighter-rouge">JSX</code>: React的开发语言,混合HTML和JS,<code class="highlighter-rouge">&lt;</code>开头用HTML解析,<code class="highlighter-rouge">{</code>用JS解析。</p> <h3 id="渲染与事件">渲染与事件</h3> <p><a href="https://reactjs.org/docs/components-and-props.html">props</a>: DOM的属性; <br /> state: model存储,保存当前的状态;</p> <p>组件包括 mount(渲染到dom tree)-&gt;update-&gt;unmount</p> <p>componentWillMount: render 之前。预处理操作,一般没啥用。</p> <blockquote> <p>The componentDidMount() hook runs after the component output has been rendered to the DOM.</p> </blockquote> <p>render: 渲染函数。</p> <p>componentDidMount: render之后,</p> <blockquote> <p>componentDidMount() is invoked immediately after a component is mounted. Initialization that requires DOM nodes should go here. If you need to load data from a remote endpoint, this is a good place to instantiate the network request.</p> </blockquote> <p>componentWillReceiveProps:</p> <blockquote> <p>componentWillReceiveProps() is invoked before a mounted component receives new props. If you need to update the state in response to prop changes (for example, to reset it), you may compare this.props and nextProps and perform state transitions using this.setState() in this method.</p> </blockquote> <p>setState:</p> <blockquote> <p>setState() enqueues changes to the component state and tells React that this component and its children need to be re-rendered with the updated state. This is the primary method you use to update the user interface in response to event handlers and server responses.</p> </blockquote> <h3 id="redux">redux</h3> <p>store: 对应props,用来共享状态。 <br /> state: store中对应的数据。 <br /> action: view的操作对应的通知类型,用于改变state, store.dispatch 来发出action。 <br /> reducer:处理action产生新的state.</p> <h3 id="参考资料">参考资料</h3> <p><a href="https://developers.google.com/web/fundamentals/performance/critical-rendering-path/constructing-the-object-model?hl=zh-cn">1 浏览器渲染</a></p> 2017 2018-01-16T00:00:00+00:00 /2018/01/16/2017-2018 <p>又是一年,三十了。又是一年,有太多遗憾,有些许成长,更多的是亏欠。</p> <p>回忆下2016年立的flag:<br /> 周末的跑步。没有坚持下去(年初换工作,新工作后只坚持了2个月);<br /> 提高开车技术。买车了,开车技术有所进步;<br /> 远行一次。又一次没有完成了,2018年不能再拖了。 <br /> 学会炒股。2017年2月份开始进入股市,目前盈亏平衡。 <br /> 和朋友认真相处。旧朋友还来得及看清,新朋友没顾得上出现。 <br /> 和家人相亲相爱。对沫沫还是耐心不够,惹/气哭了好多次;对老婆关爱不够,老婆对我一脸嫌弃;对家人脾气很大。。。 <br /> 把多看的书读完。只读了一点点。。。。</p> <p>工作:年中换了工作,新工作方式发生很大变化,开始有些不适应,现在逐渐适应,做好自己,对得起这份工资,保持好奇心、求知欲,也就无憾了吧?</p> <p>年纪有大了一点。沫沫又长了一岁。现在看她健健康康的,我就很放心了。 <br /> 记得半年多前,我在清河的出租屋里想着人生是多么无趣,工作,睡觉,一张桌椅,而不是工作,回家和孩子一起玩耍,然后睡觉。半年过去,我几乎不再想这些事情,我是麻木了吗?</p> <p>2018年,我的新flag是: 跑步。跑一次10公里+;体重减20斤; 旅游。周末周边游若干次,一次远途的家庭旅行;<br /> 控制。减少手机使用量(走路、地铁上不看手机);说话不要太「刺耳」;放慢去看世界; <br /> 读书。多看 &amp; 买的书都看完。 <br /> 家庭。对孩子更加耐心,和家人、孩子认真的相处;<br /> 其他。坚持写作。炒股盈利。开车技术大幅提升。</p> uwsgi 一瞥 2017-12-28T00:00:00+00:00 /2017/12/28/uwsgi-source-code <h3 id="uwsgi-的简单使用">uwsgi 的简单使用</h3> <p>uwsgi –help 输出它的一些命令参数。我们看一下常见的参数:</p> <p>–http-socket: 绑定 UNIX/TCP socket 端口,并使用 http 协议(eg: 0.0.0.0:1234)</p> <p>–protocol: 默认绑定的协议 <br /> –processes/–workers: 开启的进程数量 <br /> –harakiri: 进程响应超时设置 <br /> –xmlconfig/–xml: 从 xml文件中获取配置 <br /> –ini: 从ini文件中获取配置 <br /> –daemonize: 后台运行 <br /> –spooler: spooler 目录</p> <p>–logto:日志文件 <br /> –wsgi-file: 加载 wsgi 文件 <br /> –module/–wsgi: 加载 uwsgi module <br /> –callable: 默认的 uwsgi callable 名称</p> <p>ok, 我们先写一个 uwsgi 的入口(bootstrap)函数:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#bootstrap.py def application(env, start_response): start_response('200 OK', [('Content-Type','text/html')]) response = ["Hello", "World"] return response </code></pre></div></div> <p>在不使用 uwsgi 时,我们先用 <a href="https://docs.python.org/2/library/wsgiref.html#module-wsgiref.simple_server">wsgiref</a>来实现看下一个简单的应用怎么写:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>httpd = make_server('', 9090, application) print "Serving on port 9090..." httpd.serve_forever() </code></pre></div></div> <p>上面的 <code class="highlighter-rouge">application</code> 和 <code class="highlighter-rouge">wsgi server</code> 共同构成了 wsgi 的规范(即,一个 <a href="http://wsgi.tutorial.codepoint.net/application-interface">wsgi</a>是一个连接规范,just an interface specification by which server and application communicate)</p> <p>我们使用 uwsgi 来启动: <br /> <code class="highlighter-rouge">uwsgi --http :9090 --wsgi-file bootstrap.py --master --processes 4 --threads 2</code></p> <p>我们还可以通过配置 xml/ini 文件来启动:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#uwsgi.xml &lt;uwsgi&gt; &lt;http-socket&gt;0.0.0.0:9090&lt;/http-socket&gt; &lt;module&gt;bootstrap&lt;/module&gt; &lt;workers&gt;4&lt;/workers&gt; &lt;threads&gt;2&lt;/threads&gt; &lt;/uwsgi&gt; </code></pre></div></div> <p>or in ini file</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[uwsgi] http-socket = 0.0.0.0:9090 wsgi-file = bootstrap.py workers = 4 threads = 2 </code></pre></div></div> <h4 id="入口">入口</h4> <p>我们来看上面写的 application, 参数有两个 <code class="highlighter-rouge">env</code> 和 <code class="highlighter-rouge">start_response</code>, 这两个是什么东西呢?</p> <p><code class="highlighter-rouge">env</code>: 一个大 DICT 类型,里面包括一次请求的环境:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SCRIPT_NAME REQUEST_METHOD PATH_INFO REQUEST_URI HTTP_USER_AGENT SERVER_NAME/SERVER_PORT/ REMOTE_ADDR ... </code></pre></div></div> <p><code class="highlighter-rouge">start_response</code> 是一个 <a href="https://github.com/unbit/uwsgi/blob/master/plugins/python/python_plugin.c#L227">uwsgi_split</a> built-in function.</p> <h3 id="uwsgi-定时器">uwsgi 定时器</h3> <p>写一个定时任务:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from uwsgidecorators import * @timer(3) def three_seconds(signum): print("3 seconds elapsed") </code></pre></div></div> <p>加载这个定时任务:</p> <p><code class="highlighter-rouge">uwsgi --http :9090 --wsgi-file bootstrap.py --master --processes 4 --threads 2 --import mytasks.py</code></p> <p>ok, 我们能看到每个 3秒一次输出。 <br /> 或者<a href="http://uwsgi-docs-zh.readthedocs.io/zh_CN/latest/Signals.html">这样</a>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import uwsgi def hello_timer(num): print "2 seconds elapsed, signal %d raised" % num def oneshot_timer(num): print "40 seconds elapsed, signal %d raised. You will never see me again." % num uwsgi.register_signal(26, "worker", hello_timer) uwsgi.register_signal(30, "", oneshot_timer) uwsgi.add_timer(26, 2) # never-ending timer every 2 seconds uwsgi.add_rb_timer(30, 40, 1) # one shot rb timer after 40 seconds </code></pre></div></div> <p>定时器有哪些使用场景呢?</p> <ul> <li>定期 dump/上报数据</li> <li>定期 更新内存数据…</li> </ul> <h3 id="uwsgi-源码一瞥">uwsgi 源码一瞥</h3> <p>uwsgi 是一个用 c 写的 webserver, 经典的 master/worker 进程模式。</p> <blockquote> <p>一开始进入main函数,启动这个就是主进程了,uwsgi_setup函数(主进程)里面针对选项参数做一些处理,执行环境设置,执行一些hook,语言环境初始化(python),如果没有设置延迟加载app,则app在主进程加载;如果设置了延迟加载,则在每一个worker进程中都会加载一次。uwsgi_setup函数还执行了插件的初始化(eg http、python:http是gateway类型的插件,这种插件是向外暴露http服务的,python的 requests 类型的插件,用来服务请求的)、tcp socket的绑定与监听(这个是指http与work之间的通信,且它的端口是自动产生的)。最后uwsgi主进程fork了指定worker进程用来接收(accept)请求。虽然在setup中就fork了子进程,但是现在还没有开始accept。 <br /> 主进程执行部分是一个无限循环,他可以执行特定的hook以及接收信号等,总之是用来管理worker进程以及一些定时或者事件触发任务。worker部分:注册型号处理函数,执行一些hook,循环接收(accept)请求。在启动woker时可以根据–threads参数指定要产生的线程个数,否则只在当前进程启动一个线程。这些线程循环接收请求并处理。 <br /> worker中接收请求的函数wsgi_req_accept有一个锁:thunder_lock,这个锁用来串行化accept,防止“惊群”现象:主进程绑定并监听socket,然后调用fork,在各个子进程进行accept。无论任何时候,只要有一个连接尝试连接,所有的子进程都将被唤醒,但只有一个会连接成功,其他的会得到一个EAGAIN的错误,导致巨大的CPU资源浪费,如果在进程中使用线程,这个问题被再度放大。一个解决方法是串行化accept,在accept前防止一个锁。</p> </blockquote> <p><a href="https://github.com/unbit/uwsgi/blob/master/core/uwsgi.c#L2107">init</a>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#ifdef UWSGI_AS_SHARED_LIBRARY int uwsgi_init(int argc, char *argv[], char *envp[]) { #else int main(int argc, char *argv[], char *envp[]) { #endif uwsgi_setup(argc, argv, envp); return uwsgi_run(); } </code></pre></div></div> <p><a href="https://github.com/unbit/uwsgi/blob/master/core/uwsgi.c#L2127">setup</a>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>void uwsgi_setup(int argc, char *argv[], char *envp[]) { init_magic_table(uwsgi.magic_table); // initialize schemes uwsgi_setup_schemes(); // initialize the clock uwsgi_register_clock(&amp;uwsgi_unix_clock); uwsgi_set_clock("unix"); // fallback config atexit(uwsgi_fallback_config); // manage/flush logs atexit(uwsgi_flush_logs); // clear sockets, pidfiles... atexit(vacuum); // call user scripts atexit(uwsgi_exec_atexit); // call plugin specific exit hooks atexit(uwsgi_plugins_atexit); ... // call here to allows plugin to override hooks uwsgi_register_base_hooks(); uwsgi_register_logchunks(); uwsgi_log_encoders_register_embedded(); //initialize embedded plugins UWSGI_LOAD_EMBEDDED_PLUGINS // register base metrics (so plugins can override them) uwsgi_metrics_collectors_setup(); ... uwsgi_start((void *) uwsgi.argv); } </code></pre></div></div> <p><a href="https://github.com/unbit/uwsgi/blob/master/core/uwsgi.c#L3435">uwsgi_start</a>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>uwsgi_hooks_run(uwsgi.hook_in_jail, "in-jail", 1); // initialize the exception handlers uwsgi_exception_setup_handlers(); // initialize socket protocols (do it after caching !!!) uwsgi_protocols_register(); if (uwsgi.has_threads) { if (uwsgi.threads &gt; 1) uwsgi.current_wsgi_req = threaded_current_wsgi_req; (void) pthread_attr_init(&amp;uwsgi.threads_attr); if (uwsgi.threads_stacksize) { if (pthread_attr_setstacksize(&amp;uwsgi.threads_attr, uwsgi.threads_stacksize * 1024) == 0) { uwsgi_log("threads stack size set to %luk\n", (unsigned long) uwsgi.threads_stacksize); } else { uwsgi_log("!!! unable to set requested threads stacksize !!!\n"); } } pthread_mutex_init(&amp;uwsgi.lock_static, NULL); // again check for workers/sockets... if (uwsgi.sockets || uwsgi.master_process || uwsgi.no_server || uwsgi.command_mode || uwsgi.loop) { for (i = 0; i &lt; 256; i++) { if (uwsgi.p[i]-&gt;enable_threads) uwsgi.p[i]-&gt;enable_threads(); } } } uwsgi_worker_run(); </code></pre></div></div> <p><a href="https://github.com/unbit/uwsgi/blob/master/core/uwsgi.c#L3578">uwsgi_worker_run</a>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>if (uwsgi.lazy || uwsgi.lazy_apps) { uwsgi_init_all_apps(); } </code></pre></div></div> <h3 id="uwsgi-插件实现">uwsgi 插件实现</h3> <blockquote> <p>uwsgi使用的是插件化的开发方式。启动时,加载相关的若干个插件(比如python请求插件),并对这些插件进行初始化;运行时,根据输入,调用相应的插件的钩子函数,来完成处理。uwsgi只需要完成主流程和通用功能,而将具体实现交给插件去做。<br /> uwsgi采用的是经典的 单master/多worker 并发模型。master负责信号处理,初始化,创建并管理worker进程,以及绑定监听sockets。worker的每个线程,处在这样的一个循环中:accept -&gt; recv -&gt; handle request -&gt; send response。</p> </blockquote> <p><a href="https://github.com/unbit/uwsgi/blob/master/plugins/python/python_plugin.c#L2047">python plugin</a></p> <p>我们看官方的<a href="https://github.com/unbit/uwsgi-docs/blob/master/tutorials/WritingPlugins.rst">插件例子</a></p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#include &lt;uwsgi.h&gt; static int foo_init() { uwsgi_log("Hello World\n"); return 0; } struct uwsgi_plugin foobar_plugin = { .name = "foobar", .init = foo_init, }; </code></pre></div></div> <p>编译:</p> <p>python uwsgiconfig.py –plugin foobar.c</p> <p>启动:</p> <p>uwsgi –plugin ./foobar_plugin.so –http :9090 –wsgi-file bootstrap.py</p> <p>ok,这样就可以看到加载了插件的 uwsgi</p> <h3 id="其他">其他</h3> <p>官方例子中<a href="https://gist.github.com/myersguo/ea12aa987c888431ad097bb18ff06b64">c python module</a></p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcc spammify.c -fPIC -I/usr/include/python2.7 -shared -o spam.so </code></pre></div></div> <h3 id="参考">参考</h3> <p>http://wsgi.tutorial.codepoint.net/application-interface<br /> https://docs.python.org/2/extending/extending.html<br /> https://github.com/enthought/Python-2.7.3/blob/master/Demo/embed/demo.c <br /> https://www.zouyesheng.com/python-module-c.html<br /> https://www.letiantian.me/2015-09-10-understand-python-wsgi/ <br /> http://uwsgi-docs-zh.readthedocs.io/zh_CN/latest/PythonDecorators.html http://uwsgi-docs.readthedocs.io/en/latest/PythonDecorators.html</p>