Most Recent Article...

Build Your Own Go FTL Middleware

It is easy to build your own middleware for the Go-FTL server.

Note: the final version of this code is available in the download files with this blog posting. See Special.go and Special_Test.go. They are also included at the bottom of the posting.

The first step is to checkout and build the Go-FTL server.

    $ git clone https://github.com/pschlup/Go-FTL.git
    $ cd Go-FTL/server/goftl
    $ go get 
    $ go build

Now lets copy some go files so that we can make a custom version. First we will make a directory for our custom version of the main program. Let's make our server called myserver. Assuming that you are still in the Go-FTL/server/goftl directory: Assuming that you are mrexample we will build accordingly. Let's build a middleware that adds latency to requests and simulates running an application over a slow Internet connection.

    $ mkdir -p ~/go/src/mrexample/Go-FTL-Custom/myserver
    $ mkdir -p ~/go/src/mrexample/Go-FTL-Custom/middleware/Special
    $ cp *.go  ~/go/src/mrexample/Go-FTL-Custom/myserver

Let's make a copy of the DumpRequest middleware and modify that to meat our needs.

    $ cd ../midlib/DumpRequest
    $ cp *.go  ~/go/src/mrexample/Go-FTL-Custom/middleware/Special
    $ cd ~/go/src/mrexample/Go-FTL-Custom/middleware/Special

Rename files

    $ mv dumpreq.go Special.go
    $ mv dumpreq_test.go Special_test.go

Special.go will have some things that have to be changed. Let's take a look at the code and start making changes. First Replace DumpRequest with Special in both .go files.

//
// Go-FTL
//
// Copyright (C) Philip Schlump, 2014-2016
//
// Do not remove the following lines - used in auto-update.
// Version: 0.5.9
// BuildNo: 1811
// FileId: 1234
//

//
// Package Dump Request print out the request.
//

package DumpRequest

import (
	"fmt"
	"net/http"
	"os"

	"github.com/Sirupsen/logrus"
	"github.com/pschlump/Go-FTL/server/cfg"
	"github.com/pschlump/Go-FTL/server/goftlmux"
	"github.com/pschlump/Go-FTL/server/lib"
	"github.com/pschlump/Go-FTL/server/mid"
	"github.com/pschlump/godebug"
)

// --------------------------------------------------------------------------------------------------------------------------
func init() {

	// normally identical
	initNext := func(next http.Handler, g_cfg *cfg.ServerGlobalConfigType, pp_cfg interface{}, serverName string, pNo int) (rv http.Handler, err error) {
		p_cfg, ok := pp_cfg.(*DumpRequestType)
		if ok {
			p_cfg.SetNext(next)
			rv = p_cfg
		} else {
			err = mid.FtlConfigError
			logrus.Errorf("Invalid type passed at: %s", godebug.LF())
		}
		return
	}

	// normally identical
	createEmptyType := func() interface{} { return &DumpRequestType{} }

	postInit := func(h interface{}, callNo int) error {

		hh, ok := h.(*DumpRequestType)
		if !ok {
			// rw.Log.Warn(fmt.Sprintf("Error: Wrong data type passed, Line No:%d\n", hh.LineNo))
			fmt.Printf("Error: Wrong data type passed, Line No:%d\n", hh.LineNo)
			return mid.ErrInternalError
		} else {
			if hh.FileName != "" {
				var err error
				hh.outputFile, err = lib.Fopen(hh.FileName, "a")
				if err != nil {
					fmt.Printf("Error: Unable to open %s for append, Error: %s Line No:%d\n", hh.FileName, err, hh.LineNo)
					return mid.ErrInternalError
				}
			}
			hh.final, _ = lib.ParseBool(hh.Final)
		}

		return nil
	}

	cfg.RegInitItem2("DumpRequest", initNext, createEmptyType, postInit, `{
		"Paths":        { "type":["string","filepath"], "isarray":true, "default":"/" },
		"Msg":          { "type":[ "string" ], "default":"" },
		"Final":        { "type":[ "string" ], "default":"no" },
		"FileName":     { "type":[ "string","filepath" ], "default":"" },
		"LineNo":       { "type":[ "int" ], "default":"1" }
		}`)
}

// normally identical
func (hdlr *DumpRequestType) SetNext(next http.Handler) {
	hdlr.Next = next
}

var _ mid.GoFTLMiddleWare = (*DumpRequestType)(nil)

// --------------------------------------------------------------------------------------------------------------------------

type DumpRequestType struct {
	Next       http.Handler
	Paths      []string
	Msg        string
	Final      string
	FileName   string
	LineNo     int
	outputFile *os.File
	final      bool // t/f converted version of .Final
}

// Parameterized for testing? or just change the test
func NewDumpRequestServer(n http.Handler, p []string, m string, fn string) *DumpRequestType {
	return &DumpRequestType{Next: n, Paths: p, Msg: m, FileName: fn}
}

func (hdlr *DumpRequestType) ServeHTTP(www http.ResponseWriter, req *http.Request) {
	if pn := lib.PathsMatchN(hdlr.Paths, req.URL.Path); pn >= 0 {
		if rw, ok := www.(*goftlmux.MidBuffer); ok {

			trx := mid.GetTrx(rw)
			trx.PathMatched(1, "DumpRequest", hdlr.Paths, pn, req.URL.Path)

			fmt.Fprintf(hdlr.outputFile, "DumpRequest %s = %s\n", hdlr.Msg, lib.SVarI(req))
			// if hdlr.Final == "no" {
			if !hdlr.final {
				hdlr.Next.ServeHTTP(rw, req)
			} else {
				fmt.Fprintf(rw, "%s\n", lib.SVarI(req))
			}
		} else {
			rw.Log.Warn(fmt.Sprintf("Error: DumpRequest: Line No:%d  %s\n", hdlr.LineNo, mid.ErrNonMidBufferWriter))
			www.WriteHeader(http.StatusInternalServerError)
		}
	} else {
		hdlr.Next.ServeHTTP(www, req)
	}
}

/* vim: set noai ts=4 sw=4: */

By Line Numbers

1…14 - You probably want to change the comments at the top. It's going to be your code now.

58…66 - If you don't need to append to an output file then delete these lines. This section runs at initialization time and has access to the configuration information. Line 66 is a good example. It converts an input "yes"/"no" to a boolean true value. We will need to convert our input from milliseconds to a time.Duration in go.

        hh.slowDown = time.Duration(int64(hh.SlowDown)) * time.Millisecond

74…76 - You will need to to change these lines. This is the specification for the input/configuration and the type checking on that configuration. You will want your own configuration for Special. Leave Paths and LineNo in the set of values. Add in:

        "SlowDown":     { "type":[ "int" ], "default":"500" },

93…95 - Change these to match by name/type the values you have chosen at 74…76.

        SlowDown int
        ...
        slowDown time.Duration

97…98 - If you don't need them then remove these.

102…103 - This function is only used in the test code. Let's change it to take our time and convert it.

    func NewSpecialServer(n http.Handler, p []string, slow int) *SpecialType {
        return &SpecialType{
            Next:     n,
            Paths:    p,
            SlowDown: slow,
            slowDown: time.Duration(int64(slow)) * time.Millisecond,
        }
    }

113…119 - This is where your middleware performs it's work. Change this section to do something Special.

            time.Sleep(hdlr.slowDown)
            hdlr.Next.ServeHTTP(rw, req)
            return

We will also want to chagne how it returns from this function. In total the new handler looks like this:

func (hdlr *SpecialType) ServeHTTP(www http.ResponseWriter, req *http.Request) {
    if pn := lib.PathsMatchN(hdlr.Paths, req.URL.Path); pn >= 0 {
        if rw, ok := www.(*goftlmux.MidBuffer); ok {

            trx := mid.GetTrx(rw)
            trx.PathMatched(1, "Special", hdlr.Paths, pn, req.URL.Path)

            time.Sleep(hdlr.slowDown)
            hdlr.Next.ServeHTTP(rw, req)
            return

        }
    }
    hdlr.Next.ServeHTTP(www, req)
}

Test

Change the test code to validate that your middleware works.

    func Test_SpecialServer(t *testing.T) {
        tests := []struct {
            url          string
            expectedCode int
            expectSlow   bool
        }{
            {
                "http://example.com/foo?abc=def",
                http.StatusOK,
                false,
            },
            {
                "http://example.com/index.html",
                http.StatusOK,
                true,
            },
        }

        bot := mid.NewServer()
        var err error
        lib.SetupTestCreateDirs()
        ms := NewSpecialServer(bot, []string{"/index.html"}, 2000)

        fmt.Printf("Expect test to take serveral seconds\n")

        for ii, test := range tests {

            rec := httptest.NewRecorder()

            wr := goftlmux.NewMidBuffer(rec, nil)

            var req *http.Request

            req, err = http.NewRequest("GET", test.url, nil)
            if err != nil {
                t.Fatalf("Test %d: Could not create HTTP request: %v", ii, err)
            }
            lib.SetupTestMimicReq(req, "example.com")

            if db4 {
                fmt.Printf("{\"req\":%s,\n\"wr\":%s}\n", lib.SVarI(req), lib.SVarI(wr))
            }

            start := time.Now()

            if db4 {
                fmt.Fprintf(os.Stderr, "bef: %s\n", test.url)
            }
            ms.ServeHTTP(wr, req)
            if db4 {
                fmt.Fprintf(os.Stderr, "aft: %s\n", test.url)
            }

            elapsed := time.Since(start)

            if test.expectSlow {
                if elapsed < ((2000 - 1) * time.Millisecond) {
                    t.Errorf("Error %2d, did not slow request down got: %.4f seconds, expected %.4f\n", ii, elapsed.Seconds(), (2000 * time.Millisecond).Seconds())
                }
            }

            // Tests to perform on final recorder data.
            if wr.StatusCode != test.expectedCode {
                t.Errorf("Error %2d, reject error got: %d, expected %d\n", ii, wr.StatusCode, test.expectedCode)
            }

        }

    }

Now test it

    $ go test

If all went well you should see that wonderful message "PASS".

Build the Server

Include your new middleware in the main program and build.

    $ cd ../../myserver
    $ vi inc.go

Add your Special middleware to the set in the file.

    _ "mrexample/Go-FTL-Custom/middleware/Special"

Compile and final testing…

    $ go build

You should now setup a server and test it. A configuration using Special.

    {
        "test_of_Special": { "LineNo":__LINE__,
            "listen_to":[ "http://localhost:16023", "http://dev2.test1.com:16023" ],
            "plugins":[
                { "Special": { "LineNo":__LINE__,
                    "Paths":["/slow.html"],
                    "SlowDown": 2000
                } },
                { "file_server": { "LineNo":__LINE__, "Root":"./www", "Paths":"/"  } }
            ]
        }
    }

You now have a customized server with it's one unique middleware and capabilities.

Create a ./www directory with ./www/slow.html and ./www/regular.html in it.

Give it a try in your favorite browser with http://localhost:16023/slow.html and see what happens. Then try http://localhost:16023/regular.html. If you want to know how your web application behaves on an average mobile network then set "SlowDown": 114 and test. If you want to know what it is like where I live in rural Wyoming, then try 528.

Final Code

Special.go

//
// Go-FTL - Speical
//

//
// Package Special - add latency to requests to simulate slow networks.
//

package Special

import (
	"fmt"
	"net/http"
	"time"

	"github.com/Sirupsen/logrus"
	"github.com/pschlump/Go-FTL/server/cfg"
	"github.com/pschlump/Go-FTL/server/goftlmux"
	"github.com/pschlump/Go-FTL/server/lib"
	"github.com/pschlump/Go-FTL/server/mid"
	"github.com/pschlump/godebug"
)

// --------------------------------------------------------------------------------------------------------------------------
func init() {

	// normally identical
	initNext := func(next http.Handler, g_cfg *cfg.ServerGlobalConfigType, pp_cfg interface{}, serverName string, pNo int) (rv http.Handler, err error) {
		p_cfg, ok := pp_cfg.(*SpecialType)
		if ok {
			p_cfg.SetNext(next)
			rv = p_cfg
		} else {
			err = mid.FtlConfigError
			logrus.Errorf("Invalid type passed at: %s", godebug.LF())
		}
		return
	}

	// normally identical
	createEmptyType := func() interface{} { return &SpecialType{} }

	postInit := func(h interface{}, callNo int) error {

		hh, ok := h.(*SpecialType)
		if !ok {
			// rw.Log.Warn(fmt.Sprintf("Error: Wrong data type passed, Line No:%d\n", hh.LineNo))
			fmt.Printf("Error: Wrong data type passed, Line No:%d\n", hh.LineNo)
			return mid.ErrInternalError
		} else {
			hh.slowDown = time.Duration(int64(hh.SlowDown)) * time.Millisecond
		}

		return nil
	}

	cfg.RegInitItem2("Special", initNext, createEmptyType, postInit, `{
		"Paths":        { "type":["string","filepath"], "isarray":true, "default":"/" },
		"SlowDown":     { "type":[ "int" ], "default":"500" },
		"LineNo":       { "type":[ "int" ], "default":"1" }
		}`)
}

// normally identical
func (hdlr *SpecialType) SetNext(next http.Handler) {
	hdlr.Next = next
}

var _ mid.GoFTLMiddleWare = (*SpecialType)(nil)

// --------------------------------------------------------------------------------------------------------------------------

type SpecialType struct {
	Next     http.Handler
	Paths    []string
	SlowDown int
	LineNo   int
	slowDown time.Duration
}

// Parameterized for testing? or just change the test
func NewSpecialServer(n http.Handler, p []string, slow int) *SpecialType {
	return &SpecialType{
		Next:     n,
		Paths:    p,
		SlowDown: slow,
		slowDown: time.Duration(int64(slow)) * time.Millisecond,
	}
}

func (hdlr *SpecialType) ServeHTTP(www http.ResponseWriter, req *http.Request) {
	if pn := lib.PathsMatchN(hdlr.Paths, req.URL.Path); pn >= 0 {
		if rw, ok := www.(*goftlmux.MidBuffer); ok {

			trx := mid.GetTrx(rw)
			trx.PathMatched(1, "Special", hdlr.Paths, pn, req.URL.Path)

			time.Sleep(hdlr.slowDown)
			hdlr.Next.ServeHTTP(rw, req)
			return

		}
	}
	hdlr.Next.ServeHTTP(www, req)
}

/* vim: set noai ts=4 sw=4: */

Special test code.

//
// Go-FTL - Speical
//

package Special

import (
	"fmt"
	"net/http"
	"net/http/httptest"
	"os"
	"testing"
	"time"

	"github.com/pschlump/Go-FTL/server/goftlmux"
	"github.com/pschlump/Go-FTL/server/lib"
	"github.com/pschlump/Go-FTL/server/mid"
)

func Test_SpecialServer(t *testing.T) {
	tests := []struct {
		url          string
		expectedCode int
		expectSlow   bool
	}{
		{
			"http://example.com/foo?abc=def",
			http.StatusOK,
			false,
		},
		{
			"http://example.com/index.html",
			http.StatusOK,
			true,
		},
	}

	bot := mid.NewServer()
	var err error
	lib.SetupTestCreateDirs()
	ms := NewSpecialServer(bot, []string{"/index.html"}, 2000)

	fmt.Printf("Expect test to take serveral seconds\n")

	for ii, test := range tests {

		rec := httptest.NewRecorder()

		wr := goftlmux.NewMidBuffer(rec, nil)

		var req *http.Request

		req, err = http.NewRequest("GET", test.url, nil)
		if err != nil {
			t.Fatalf("Test %d: Could not create HTTP request: %v", ii, err)
		}
		lib.SetupTestMimicReq(req, "example.com")

		if db4 {
			fmt.Printf("{\"req\":%s,\n\"wr\":%s}\n", lib.SVarI(req), lib.SVarI(wr))
		}

		start := time.Now()

		if db4 {
			fmt.Fprintf(os.Stderr, "bef: %s\n", test.url)
		}
		ms.ServeHTTP(wr, req)
		if db4 {
			fmt.Fprintf(os.Stderr, "aft: %s\n", test.url)
		}

		elapsed := time.Since(start)

		if test.expectSlow {
			if elapsed < ((2000 - 1) * time.Millisecond) {
				t.Errorf("Error %2d, did not slow request down got: %.4f seconds, expected %.4f\n", ii, elapsed.Seconds(), (2000 * time.Millisecond).Seconds())
			}
		}

		// Tests to perform on final recorder data.
		if wr.StatusCode != test.expectedCode {
			t.Errorf("Error %2d, reject error got: %d, expected %d\n", ii, wr.StatusCode, test.expectedCode)
		}

	}

}

const db4 = false

/* vim: set noai ts=4 sw=4: */

Modified for the main program:

//
// Go-FTL
//
// Copyright (C) Philip Schlump, 2014-2016
//
// Do not remove the following lines - used in auto-update.
// Version: 0.5.9
// BuildNo: 1811
// FileId: 1122
//

package main

import (
	_ "github.com/pschlump/Go-FTL/server/midlib/AesSrp"
	_ "github.com/pschlump/Go-FTL/server/midlib/BasicAuth"
	_ "github.com/pschlump/Go-FTL/server/midlib/BasicAuthPgSql"
	_ "github.com/pschlump/Go-FTL/server/midlib/BasicAuthRedis"
	_ "github.com/pschlump/Go-FTL/server/midlib/Cookie"
	_ "github.com/pschlump/Go-FTL/server/midlib/DirectoryBrowse"
	_ "github.com/pschlump/Go-FTL/server/midlib/DumpRequest"
	_ "github.com/pschlump/Go-FTL/server/midlib/DumpResponse"
	_ "github.com/pschlump/Go-FTL/server/midlib/Echo"
	_ "github.com/pschlump/Go-FTL/server/midlib/Else"
	_ "github.com/pschlump/Go-FTL/server/midlib/GeoIpFilter" // move this out to github.com/pschlump/Go-FTL-GeoIpFilter
	_ "github.com/pschlump/Go-FTL/server/midlib/GoTemplate"
	_ "github.com/pschlump/Go-FTL/server/midlib/Gzip"
	_ "github.com/pschlump/Go-FTL/server/midlib/HTML5Path"
	_ "github.com/pschlump/Go-FTL/server/midlib/Header"
	_ "github.com/pschlump/Go-FTL/server/midlib/InMemoryCache"
	_ "github.com/pschlump/Go-FTL/server/midlib/JSONToTable"
	_ "github.com/pschlump/Go-FTL/server/midlib/JSONp"
	_ "github.com/pschlump/Go-FTL/server/midlib/Latency"
	_ "github.com/pschlump/Go-FTL/server/midlib/LimitExtensionTo"
	_ "github.com/pschlump/Go-FTL/server/midlib/LimitPathReTo"
	_ "github.com/pschlump/Go-FTL/server/midlib/LimitPathTo"
	_ "github.com/pschlump/Go-FTL/server/midlib/Logging"
	_ "github.com/pschlump/Go-FTL/server/midlib/LoginRequired"
	_ "github.com/pschlump/Go-FTL/server/midlib/Minify"
	_ "github.com/pschlump/Go-FTL/server/midlib/Redirect"
	_ "github.com/pschlump/Go-FTL/server/midlib/RedisList"
	_ "github.com/pschlump/Go-FTL/server/midlib/RedisListRaw"
	_ "github.com/pschlump/Go-FTL/server/midlib/RejectDirectory"
	_ "github.com/pschlump/Go-FTL/server/midlib/RejectExtension"
	_ "github.com/pschlump/Go-FTL/server/midlib/RejectHotlink"
	_ "github.com/pschlump/Go-FTL/server/midlib/RejectIpAddress"
	_ "github.com/pschlump/Go-FTL/server/midlib/RejectPath"
	_ "github.com/pschlump/Go-FTL/server/midlib/RejectRePath"
	_ "github.com/pschlump/Go-FTL/server/midlib/Rewrite"
	_ "github.com/pschlump/Go-FTL/server/midlib/RewriteProxy"
	_ "github.com/pschlump/Go-FTL/server/midlib/SocketIO"
	_ "github.com/pschlump/Go-FTL/server/midlib/Status"

	// _ "github.com/pschlump/m6"

	_ "github.com/pschlump/TabServer2"
	_ "github.com/pschlump/mon-alive/middleware"

	// Modules that are not in the midlib deirectory
	_ "github.com/pschlump/Go-FTL/server/fileserve"
)


  • Angular 2.0 Basics - Instalation and Setup: 2016-05-20

    Overview

    5 to 10 minutes to read and implement.

    1) Installation of Angular 2.0 1) Setting up npm and node 1) An intitial package 1) Using nodes internal server 2) Setting up TypeScript 2) Creating our first Angular 2.0 application 3) Using a non-node.js server with Angular 2.0

    Read More...

  • 2016 02 25.Why Computer Security Needs To Be Incredibly Strong: 2016-02-25

    Why Computer Security Needs to be Incredibly Strong

    When you think about your garage and security it is mostly about keeping people out so that they can't steal your stuff. The people are the ones in your vicinity. They may not be your neighbors, but at most they are from across town. That limits your exposure.

    A lot of time you just keep the more valuable stuff in a more secure location. You don't store your jewelry in the garage. You hide that inside the house with a better lock and lock the door to the house.

    Burglars might still find it but this made it difficult enough. If the government

    Read More...

  • Go: How to Iterate Over the Keys of a Map: 2013-12-05

    This is kind of embarrassing but I had to go searching in the Go documentation for this. It should have been obvious. I actually know what happened. I spent too much time using languages that have special syntax for all sorts of stuff. That left me thinking that iterating over the keys for a map would require something special. Thankfully in Go it all works the same.

    Read More...

  • Go: How to Sort the Keys on a Map: 2013-12-05

    I often want the values in a map to be in key order. That means sorting the keys for a map.

    Read More...

 

Before You Go....

Have you read "Unintend Consinsequences"?

"I laughed so hard it hurt..."
    Rod Brown

"Incredibly funny! Incredibly true!"
    Tad Stevens