First thing first, I want to list my own devices, which I have through the years:
App/Service I use daily:
The purpose is that I want my data to be safe, secure, and can be easily recovered if I lost some devices; + + +
+ + + + + + + Index +First thing first, I want to list my own devices, which I have through the years:
+App/Service I use daily:
+The purpose is that I want my data to be safe, secure, and can be easily recovered if I lost some devices; or in the worst situation, I lost all. -Because you know, it is hard to guess what is waiting for us in the future.
There are 2 sections which I want to share, the first is How to backup, the second is Recover strategy.
Before I talk about backup, I want to talk about data. -In specifically, which data should I backup?
I use Arch Linux and macOS, primarily work in the terminal so I have too many dotfiles, for example, ~/.config/nvim/init.lua.
-Each time I reinstall Arch Linux (I like it a lot), I need to reconfigure all the settings, and it is time-consuming.
So for the DE and UI settings, I keep it as default as possible, unless it's getting in my way, I leave the default setting there and forget about it. -The others are dotfiles, which I write my own dotfiles tool to backup and reconfigure easily and quickly. +Because you know, it is hard to guess what is waiting for us in the future.
+There are 2 sections which I want to share, the first is How to backup, the second is Recover strategy.
+Before I talk about backup, I want to talk about data. +In specifically, which data should I backup?
+I use Arch Linux and macOS, primarily work in the terminal so I have too many dotfiles, for example, ~/.config/nvim/init.lua.
+Each time I reinstall Arch Linux (I like it a lot), I need to reconfigure all the settings, and it is time-consuming.
So for the DE and UI settings, I keep it as default as possible, unless it's getting in my way, I leave the default setting there and forget about it. +The others are dotfiles, which I write my own dotfiles tool to backup and reconfigure easily and quickly. Also, I know that installing Arch Linux is not easy, despite I install it too many times (Like thousand times since I was in high school). -Not because it is hard, but as life goes on, the official install guide keeps getting new update and covering too many cases for my own personal use, so I write my own guide to quickly capture what I need to do. +Not because it is hard, but as life goes on, the official install guide keeps getting new update and covering too many cases for my own personal use, so I write my own guide to quickly capture what I need to do. I back up all my dotfiles in GitHub and GitLab as I trust them both. -Also as I travel the Internet, I discover Codeberg and Treehouse and use them as another backup for git repo.
So that is my dotfiles, for my regular data, like Wallpaper or Books, Images, I use Google Drive (Actually I pay for it). +Also as I travel the Internet, I discover Codeberg and Treehouse and use them as another backup for git repo.
+So that is my dotfiles, for my regular data, like Wallpaper or Books, Images, I use Google Drive (Actually I pay for it). But the step: open the webpage, click the upload button and choose files seems boring and time-consuming. So I use Rclone, it supports Google Drive, One Drive and many providers but I only use Google Drive for now. -The commands are simple:
# Sync from local to remote
+The commands are simple:
+# Sync from local to remote
rclone sync MyBooks remote:MyBooks -P --exclude .DS_Store
-# Sync from remote to local
-rclone sync remote:MyBooks MyBooks -P --exclude .DS_Store
Before you use Rclone to sync to Google Drive, you should read Google Drive rclone configuration first.
For private data, I use restic which can be used with Rclone:
# Init
+# Sync from remote to local
+rclone sync remote:MyBooks MyBooks -P --exclude .DS_Store
+Before you use Rclone to sync to Google Drive, you should read Google Drive rclone configuration first.
+For private data, I use restic which can be used with Rclone:
+# Init
restic -r rclone:remote:PrivateData init
-# Backup
+# Backup
restic -r rclone:remote:PrivateData backup PrivateData
-# Cleanup old backups
+# Cleanup old backups
restic -r rclone:remote:PrivateData forget --keep-last 1 --prune
-# Restore
-restic -r rclone:remote:PrivateData restore latest --target ~
The next data is my passwords and my OTPs.
+# Restore
+restic -r rclone:remote:PrivateData restore latest --target ~
The next data is my passwords and my OTPs. These are the things which I'm scare to lose the most. -First thing first, I enable 2-Step Verification for all of my important accounts, should use both OTP and phone method.
I use Bitwarden for passwords (That is a long story, coming from Google Password manager to Firefox Lockwise and then settle down with Bitwarden) and Aegis for OTPs. -The reason I choose Aegis, not Authy (I use Authy for so long but Aegis is definitely better) is because Aegis allows me to extract all the OTPs to a single file (Can be encrypted), which I use to transfer or backup easily.
As long as Bitwarden provides free passwords stored, I use all of its apps, extensions so that I can easily sync passwords between laptops and phones. -The thing I need to remember is the master password of Bitwarden in my head.
With Aegis, I export the data, then sync it to Google Drive, also store it locally in my phone.
The main problem here is the OTP, I can not store all of my OTPs in the cloud completely. -Because if I want to access my OTPs in the cloud, I should log in, and then input my OTP, this is a circle, my friends.
There are many strategies that I process to react as if something strange is happening to my devices.
If I lost my laptops, single laptop or all, do not panic as long as I have my phones. -The OTPs are in there, the passwords are in Bitwarden cloud, other data is in Google Drive so nothing is lost here.
If I lost my phone, but not my laptops, I use the OTPs which are stored locally in my laptops.
In the worst situation, I lost everything, my laptops, my phone. +First thing first, I enable 2-Step Verification for all of my important accounts, should use both OTP and phone method.
+I use Bitwarden for passwords (That is a long story, coming from Google Password manager to Firefox Lockwise and then settle down with Bitwarden) and Aegis for OTPs. +The reason I choose Aegis, not Authy (I use Authy for so long but Aegis is definitely better) is because Aegis allows me to extract all the OTPs to a single file (Can be encrypted), which I use to transfer or backup easily.
+As long as Bitwarden provides free passwords stored, I use all of its apps, extensions so that I can easily sync passwords between laptops and phones. +The thing I need to remember is the master password of Bitwarden in my head.
+With Aegis, I export the data, then sync it to Google Drive, also store it locally in my phone.
+The main problem here is the OTP, I can not store all of my OTPs in the cloud completely. +Because if I want to access my OTPs in the cloud, I should log in, and then input my OTP, this is a circle, my friends.
+There are many strategies that I process to react as if something strange is happening to my devices.
+If I lost my laptops, single laptop or all, do not panic as long as I have my phones. +The OTPs are in there, the passwords are in Bitwarden cloud, other data is in Google Drive so nothing is lost here.
+If I lost my phone, but not my laptops, I use the OTPs which are stored locally in my laptops.
+In the worst situation, I lost everything, my laptops, my phone. The first step is to recover my SIM, then log in to Google account using the password and SMS OTP. -After that, log in to Bitwarden account using the master password and OTP from Gmail, which I open previously.
This guide will be updated regularly I promise.
Feel free to ask me via email -Mastodon \ No newline at end of file +After that, log in to Bitwarden account using the master password and OTP from Gmail, which I open previously. +This guide will be updated regularly I promise.
+ + Feel free to ask me via email + Mastodon + + diff --git a/docs/2022-06-08-dockerfile-go.html b/docs/2022-06-08-dockerfile-go.html index 57c9076..d60862f 100644 --- a/docs/2022-06-08-dockerfile-go.html +++ b/docs/2022-06-08-dockerfile-go.html @@ -1,44 +1,97 @@ -IndexEach time I start a new Go project, I repeat many steps.
-Like set up .gitignore, CI configs, Dockerfile, ...
So I decide to have a baseline Dockerfile like this:
FROM golang:1.19-bullseye as builder
+
+
+
+
+
+
+
+
+
+ Index
+ Dockerfile for Go
+Each time I start a new Go project, I repeat many steps.
+Like set up .gitignore, CI configs, Dockerfile, ...
+So I decide to have a baseline Dockerfile like this:
+FROM golang:1.19-bullseye as builder
+
+RUN go install golang.org/dl/go1.19@latest \
&& go1.19 download
-WORKDIR /build
+WORKDIR /build
-COPY go.mod .
-COPY go.sum .
-COPY vendor .
-COPY . .
+COPY go.mod .
+COPY go.sum .
+COPY vendor .
+COPY . .
-RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOAMD64=v3 go build -o ./app -tags timetzdata -trimpath .
+RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOAMD64=v3 go build -o ./app -tags timetzdata -trimpath .
-FROM gcr.io/distroless/base-debian11
+FROM gcr.io/distroless/base-debian11
-COPY --from=builder /build/app /app
+COPY --from=builder /build/app /app
-ENTRYPOINT ["/app"]
I use multi-stage build to keep my image size small.
-First stage is Go official image,
-second stage is Distroless.
Before Distroless, I use Alpine official image,
+ENTRYPOINT ["/app"]
I use multi-stage build to keep my image size small. +First stage is Go official image, +second stage is Distroless.
+Before Distroless, I use Alpine official image, There is a whole discussion on the Internet to choose which is the best base image for Go. After reading some blogs, I discover Distroless as a small and secure base image. -So I stick with it for a while.
Also, remember to match Distroless Debian version with Go official image Debian version.
FROM golang:1.19-bullseye as builderThis is Go image I use as a build stage. -This can be official Go image or custom image is required in some companies.
RUN go install golang.org/dl/go1.19@latest \
- && go1.19 downloadThis is optional. -In my case, my company is slow to update Go image so I use this trick to install latest Go version.
WORKDIR /build
+So I stick with it for a while.
+Also, remember to match Distroless Debian version with Go official image Debian version.
+FROM golang:1.19-bullseye as builder
+This is Go image I use as a build stage.
+This can be official Go image or custom image is required in some companies.
+RUN go install golang.org/dl/go1.19@latest \
+ && go1.19 download
+This is optional.
+In my case, my company is slow to update Go image so I use this trick to install latest Go version.
+WORKDIR /build
-COPY go.mod .
-COPY go.sum .
-COPY vendor .
-COPY . .
I use /build to emphasize that I am building something in that directory.
The 4 COPY lines are familiar if you use Go enough.
+COPY go.mod .
+COPY go.sum .
+COPY vendor .
+COPY . .
I use /build to emphasize that I am building something in that directory.
The 4 COPY lines are familiar if you use Go enough.
First is go.mod and go.sum because it defines Go modules.
-The second is vendor, this is optional but I use it because I don't want each time I build Dockerfile, I need to redownload Go modules.
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOAMD64=v3 go build -o ./app -tags timetzdata -trimpath .This is where I build Go program.
CGO_ENABLED=0 because I don't want to mess with C libraries.
+The second is vendor, this is optional but I use it because I don't want each time I build Dockerfile, I need to redownload Go modules.
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOAMD64=v3 go build -o ./app -tags timetzdata -trimpath .This is where I build Go program.
+CGO_ENABLED=0 because I don't want to mess with C libraries.
GOOS=linux GOARCH=amd64 is easy to explain, Linux with x86-64.
-GOAMD64=v3 is new since Go 1.18,
-I use v3 because I read about AMD64 version in Arch Linux rfcs. TLDR's newer computers are already x86-64-v3.
-tags timetzdata to embed timezone database incase base image does not have.
--trimpath to support reproduce build.
FROM gcr.io/distroless/base-debian11
+GOAMD64=v3 is new since Go 1.18,
+I use v3 because I read about AMD64 version in Arch Linux rfcs. TLDR's newer computers are already x86-64-v3.
+-tags timetzdata to embed timezone database incase base image does not have.
+-trimpath to support reproduce build.
+FROM gcr.io/distroless/base-debian11
-COPY --from=builder /build/app /app
+COPY --from=builder /build/app /app
-ENTRYPOINT ["/app"]
Finally, I copy app to Distroless base image.
Feel free to ask me via email
-Mastodon
\ No newline at end of file
+ENTRYPOINT ["/app"]Finally, I copy app to Distroless base image.
It is hard to write bootstrap tool to quickly create Go service. + + +
+ + + + + + + Index +It is hard to write bootstrap tool to quickly create Go service. So I write this guide instead. This is a quick checklist for me every damn time I need to write a Go service from scratch. -Also, this is my personal opinion, so feel free to comment.
main.go
+Also, this is my personal opinion, so feel free to comment.
+Structure
+main.go
internal
-| business
-| | http
-| | | handler.go
-| | | service.go
-| | | models.go
-| | grpc
-| | | handler.go
-| | | models.go
-| | consumer
-| | | handler.go
-| | | service.go
-| | | models.go
-| | service.go
-| | repository.go
-| | models.go
All business codes are inside internal.
-Each business has a different directory business.
Inside each business, there are 2 handlers: http, grpc:
http is for public APIs (Android, iOS, ... are clients).grpc is for internal APIs (other services are clients).consumer is for consuming messages from queue (Kafka, RabbitMQ, ...).For each handler, there are usually 3 layers: handler, service, repository:
handler interacts directly with gRPC, REST or consumer using specific codes (cookies, ...) In case gRPC, there are frameworks outside handle for us so we can write business/logic codes here too. But remember, gRPC only.service is where we write business/logic codes, and only business/logic codes is written here.repository is where we write codes which interacts with database/cache like MySQL, Redis, ...models is where we put all request, response, data models.Location:
handler must exist inside grpc, http, consumer.service, models can exist directly inside of business if both grpc, http, consumer has same business/logic.repository should be placed directly inside of business.If we have too many services, some of the logic will be overlapped.
For example, service A and service B both need to make POST call API to service C. +| business +| | http +| | | handler.go +| | | service.go +| | | models.go +| | grpc +| | | handler.go +| | | models.go +| | consumer +| | | handler.go +| | | service.go +| | | models.go +| | service.go +| | repository.go +| | models.go
All business codes are inside internal.
+Each business has a different directory business.
Inside each business, there are 2 handlers: http, grpc:
http is for public APIs (Android, iOS, ... are clients).grpc is for internal APIs (other services are clients).consumer is for consuming messages from queue (Kafka, RabbitMQ, ...).For each handler, there are usually 3 layers: handler, service, repository:
handler interacts directly with gRPC, REST or consumer using specific codes (cookies, ...) In case gRPC, there are frameworks outside handle for us so we can write business/logic codes here too. But remember, gRPC only.service is where we write business/logic codes, and only business/logic codes is written here.repository is where we write codes which interacts with database/cache like MySQL, Redis, ...models is where we put all request, response, data models.Location:
+handler must exist inside grpc, http, consumer.service, models can exist directly inside of business if both grpc, http, consumer has same business/logic.repository should be placed directly inside of business.If we have too many services, some of the logic will be overlapped.
+For example, service A and service B both need to make POST call API to service C. If service A and service B both have libs to call service C to do that API, we need to move the libs to some common pkg libs. -So in the future, service D which needs to call C will not need to copy libs to handle service C api but only need to import from common pkg libs.
Another bad practice is adapter service. -No need to write a new service if what we need is just common pkg libs.
If I see someone using global var, I swear I will shoot them twice in the face.
Why?
For simple struct with 1 or 2 fields, no need to use functional options.
func main() {
- s := NewS(WithA(1), WithB("b"))
- fmt.Printf("%+v\n", s)
+So in the future, service D which needs to call C will not need to copy libs to handle service C api but only need to import from common pkg libs.
+Another bad practice is adapter service.
+No need to write a new service if what we need is just common pkg libs.
+Taste on style guide
+Stop using global var
+If I see someone using global var, I swear I will shoot them twice in the face.
+Why?
+For simple struct with 1 or 2 fields, no need to use functional options.
+ +func main() {
+ s := NewS(WithA(1), WithB("b"))
+ fmt.Printf("%+v\n", s)
}
-type S struct {
- fieldA int
- fieldB string
+type S struct {
+ fieldA int
+ fieldB string
}
-type OptionS func(s *S)
+type OptionS func(s *S)
-func WithA(a int) OptionS {
- return func(s *S) {
- s.fieldA = a
+func WithA(a int) OptionS {
+ return func(s *S) {
+ s.fieldA = a
}
}
-func WithB(b string) OptionS {
- return func(s *S) {
- s.fieldB = b
+func WithB(b string) OptionS {
+ return func(s *S) {
+ s.fieldB = b
}
}
-func NewS(opts ...OptionS) *S {
- s := &S{}
- for _, opt := range opts {
- opt(s)
+func NewS(opts ...OptionS) *S {
+ s := &S{}
+ for _, opt := range opts {
+ opt(s)
}
- return s
-}In above example, I construct s with WithA and WithB option.
-No need to pass direct field inside s.
If business logic involves calling too many APIs, but they are not depend on each other. -We can fire them parallel :)
Personally, I prefer errgroup to WaitGroup (https://pkg.go.dev/sync#WaitGroup).
-Because I always need deal with error.
Example:
eg, egCtx := errgroup.WithContext(ctx)
+ return s
+}In above example, I construct s with WithA and WithB option.
+No need to pass direct field inside s.
If business logic involves calling too many APIs, but they are not depend on each other. +We can fire them parallel :)
+Personally, I prefer errgroup to WaitGroup (https://pkg.go.dev/sync#WaitGroup).
+Because I always need deal with error.
Example:
+eg, egCtx := errgroup.WithContext(ctx)
-eg.Go(func() error {
- // Do some thing
- return nil
+eg.Go(func() error {
+ // Do some thing
+ return nil
})
-eg.Go(func() error {
- // Do other thing
- return nil
+eg.Go(func() error {
+ // Do other thing
+ return nil
})
-if err := eg.Wait(); err != nil {
- // Handle error
-}Please don't use external libs for WorkerPool, I don't want to deal with dependency hell.
vendorOnly need if you need something from vendor, to generate mock or something else.
build.go to include build tools in go.modTo easily control version of build tools.
For example build.go:
//go:build tools
-// +build tools
+if err := eg.Wait(); err != nil {
+ // Handle error
+}Please don't use external libs for WorkerPool, I don't want to deal with dependency hell.
+vendor
+Only need if you need something from vendor, to generate mock or something else.
build.go to include build tools in go.modTo easily control version of build tools.
+For example build.go:
//go:build tools
+// +build tools
-package main
+package main
-import (
- _ "github.com/golang/protobuf/protoc-gen-go"
-)And then in Makefile:
build:
- go install github.com/golang/protobuf/protoc-gen-goWe always get the version of build tools in go.mod each time we install it.
-Future contributors will not cry anymore.
What is the point to pass many params (do-it, --abc, --xyz) when what we only need is start service?
In my case, service starts with only config, and config should be read from file or environment like The Twelve Factors guide.
Just don't.
Use protocolbuffers/protobuf-go, grpc/grpc-go for gRPC.
Write 1 for both gRPC, REST sounds good, but in the end, it is not worth it.
prototool is deprecated, and buf can generate, lint, format as good as prototool.
Don't use gin.Context when pass context from handler layer to service layer, use gin.Context.Request.Context() instead.
It is fast!
func (*Logger) With. Because if log line is too long, there is a possibility that we can lost it.MarshalLogObject when we need to hide some field of object when log (field is long or has sensitive value)Panic. Use Fatal for errors when start service to check dependencies. If you really need panic level, use DPanic.zap.Any.contextID or traceID in every log lines for easily debug.Only init config in main or cmd layer.
-Do not use viper.Get... in business layer or inside business layer.
Why?
Also, be careful if config value is empty. -You should decide to continue or stop the service if there is no config.
Each ORM libs has each different syntax. +import ( + _ "github.com/golang/protobuf/protoc-gen-go" +)
And then in Makefile:
build:
+ go install github.com/golang/protobuf/protoc-gen-goWe always get the version of build tools in go.mod each time we install it.
+Future contributors will not cry anymore.
What is the point to pass many params (do-it, --abc, --xyz) when what we only need is start service?
In my case, service starts with only config, and config should be read from file or environment like The Twelve Factors guide.
+Just don't.
+Use protocolbuffers/protobuf-go, grpc/grpc-go for gRPC.
+Write 1 for both gRPC, REST sounds good, but in the end, it is not worth it.
+prototool is deprecated, and buf can generate, lint, format as good as prototool.
+Don't use gin.Context when pass context from handler layer to service layer, use gin.Context.Request.Context() instead.
It is fast!
+func (*Logger) With. Because if log line is too long, there is a possibility that we can lost it.MarshalLogObject when we need to hide some field of object when log (field is long or has sensitive value)Panic. Use Fatal for errors when start service to check dependencies. If you really need panic level, use DPanic.zap.Any.contextID or traceID in every log lines for easily debug.Only init config in main or cmd layer.
+Do not use viper.Get... in business layer or inside business layer.
Why?
+Also, be careful if config value is empty. +You should decide to continue or stop the service if there is no config.
+Each ORM libs has each different syntax. To learn and use those libs correctly is time consuming. So just stick to plain SQL. -It is easier to debug when something is wrong.
But database/sql has its own limit.
+It is easier to debug when something is wrong.
But database/sql has its own limit.
For example, it is hard to get primary key after insert/update.
So may be you want to use ORM for those cases.
-I hear that go-gorm/gorm, ent/ent is good.
It is easy to write a suite test, thanks to testify. +I hear that go-gorm/gorm, ent/ent is good.
+It is easy to write a suite test, thanks to testify. Also, for mocking, there are many options out there. -Pick 1 then sleep peacefully.
The first is easy to use but not powerful as the later. -If you want to make sure mock func is called with correct times, use the later.
Example with matryer/moq:
// Only gen mock if source code file is newer than mock file
-// https://jonwillia.ms/2019/12/22/conditional-gomock-mockgen
-//go:generate sh -c "test service_mock_generated.go -nt $GOFILE && exit 0; moq -rm -out service_mock_generated.go . Service"Don't cast proto enum:
// Bad
-a := cast.ToInt32(servicev1.ReasonCode_ABC)
+Pick 1 then sleep peacefully.
+If need to mock, choose matryer/moq or golang/mock
+
+The first is easy to use but not powerful as the later.
+If you want to make sure mock func is called with correct times, use the later.
+Example with matryer/moq:
+// Only gen mock if source code file is newer than mock file
+// https://jonwillia.ms/2019/12/22/conditional-gomock-mockgen
+//go:generate sh -c "test service_mock_generated.go -nt $GOFILE && exit 0; moq -rm -out service_mock_generated.go . Service"
+Be careful with spf13/cast
+
+Don't cast proto enum:
+// Bad
+a := cast.ToInt32(servicev1.ReasonCode_ABC)
-// Good
-a := int32(servicev1.ReasonCode_ABC)
Use stringer if you want your type enum can be print as string
type Drink int
+// Good
+a := int32(servicev1.ReasonCode_ABC)
+Use stringer if you want your type enum can be print as string
+type Drink int
-const (
- Beer Drink = iota
- Water
- OrangeJuice
-)
go install golang.org/x/tools/cmd/stringer@latest
+const (
+ Beer Drink = iota
+ Water
+ OrangeJuice
+)
+go install golang.org/x/tools/cmd/stringer@latest
-# Run inside directory which contains Drink
-stringer -type=DrinkDon't waste your time rewrite rate limiter if your use case is simple, use rate or go-redis/redis_rate
rate if you want rate limiter locally in your single instance of service.
-redis_rate if you want rate limiter distributed across all your instances of service.
Replace go fmt, goimports with mvdan/gofumpt.
gofumpt provides more rules when format Go codes.
Use golangci/golangci-lint.
No need to say more.
-Lint or get the f out!
If you get fieldalignment error, use fieldalignment to fix them.
# Install
+# Run inside directory which contains Drink
+stringer -type=Drink
+Don't waste your time rewrite rate limiter if your use case is simple, use rate or go-redis/redis_rate
+
+rate if you want rate limiter locally in your single instance of service.
+redis_rate if you want rate limiter distributed across all your instances of service.
+Replace go fmt, goimports with mvdan/gofumpt.
+gofumpt provides more rules when format Go codes.
+Use golangci/golangci-lint.
+No need to say more.
+Lint or get the f out!
+If you get fieldalignment error, use fieldalignment to fix them.
+# Install
go install golang.org/x/tools/go/analysis/passes/fieldalignment/cmd/fieldalignment@latest
-# Fix
-fieldalignment -fix ./internal/business/*.go
Thanks
Feel free to ask me via email
-Mastodon
\ No newline at end of file
+# Fix
+fieldalignment -fix ./internal/business/*.goThere are many use cases where we need to use a unique ID. -In my experience, I only encouter 2 cases:
In my Go universe, there are some libs to help us with this:
The ID is used only for trace and log. + + +
+ + + + + + + Index +There are many use cases where we need to use a unique ID. +In my experience, I only encouter 2 cases:
+In my Go universe, there are some libs to help us with this:
+ +The ID is used only for trace and log. If same ID is generated twice (because maybe the possibilty is too small but not 0), honestly I don't care. -When I use that ID to search log , if it pops more than things I care for, it is still no harm to me.
My choice for this use case is rs/xid. -Because it is small (not span too much on log line) and copy friendly.
Why I don't use auto increment key for primary key? +When I use that ID to search log , if it pops more than things I care for, it is still no harm to me.
+My choice for this use case is rs/xid. +Because it is small (not span too much on log line) and copy friendly.
+Why I don't use auto increment key for primary key? The answer is simple, I don't want to write database specific SQL. SQLite has some different syntax from MySQL, and PostgreSQL and so on. -Every logic I can move to application layer from database layer, I will.
In the past and present, I use google/uuid, specificially I use UUID v4. +Every logic I can move to application layer from database layer, I will.
+In the past and present, I use google/uuid, specificially I use UUID v4. In the future I will look to use segmentio/ksuid and oklog/ulid (trial and error of course). Both are sortable, but google/uuid is not. -The reason I'm afraid because the database is sensitive subject, and I need more testing and battle test proof to trust those libs.
I think about adding prefix to ID to identify which resource that ID represents.
I think about adding prefix to ID to identify which resource that ID represents.
+buf from prototoolWhy? Because prototool is outdated, and can not run on M1 mac.
We need 3 files:
build.go: need to install protoc-gen-* binaries with pin version in go.modbuf.yamlbuf.gen.yamlFYI, the libs version I use:
build.go:
//go:build tools
-// +build tools
+
+
+
+
+
+
+
+
+
+ Index
+ Migrate to buf from prototool
+
+Why? Because prototool is outdated, and can not run on M1 mac.
+We need 3 files:
+build.go: need to install protoc-gen-* binaries with pin version in go.mod
+buf.yamlbuf.gen.yamlFYI, the libs version I use:
+build.go:
//go:build tools
+// +build tools
+
+import (
+ _ "github.com/envoyproxy/protoc-gen-validate"
+ _ "github.com/golang/protobuf/protoc-gen-go"
+ _ "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway"
+ _ "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger"
+ _ "github.com/kei2100/protoc-gen-marshal-zap/plugin/protoc-gen-marshal-zap"
+)buf.yaml
version: v1
+deps:
+ - buf.build/haunt98/googleapis:b38d93f7ade94a698adff9576474ae7c
+ - buf.build/haunt98/grpc-gateway:ecf4f0f58aa8496f8a76ed303c6e06c7
+ - buf.build/haunt98/protoc-gen-validate:2686264610fc4ad4a9fcc932647e279d
+ - buf.build/haunt98/marshal-zap:2a593ca925134680a5820d3f13c1be5a
+breaking:
+ use:
+ - FILE
+lint:
+ use:
+ - DEFAULTbuf.gen.yaml:
version: v1
+plugins:
+ - name: go
+ out: pkg
+ opt:
+ - plugins=grpc
+ - name: grpc-gateway
+ out: pkg
+ opt:
+ - logtostderr=true
+ - name: swagger
+ out: .
+ opt:
+ - logtostderr=true
+ - name: validate
+ out: pkg
+ opt:
+ - lang=go
+ - name: marshal-zap
+ out: pkgUpdate Makefile:
gen:
go install github.com/golang/protobuf/protoc-gen-go
go install github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
go install github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger
@@ -46,5 +99,36 @@
go install github.com/bufbuild/buf/cmd/buf@latest
buf mod update
buf format -w
- buf generateRun make gen to have fun of course.
Remember grpc-ecosystem/grpc-gateway, envoyproxy/protoc-gen-validate, kei2100/protoc-gen-marshal-zap is optional, so feel free to delete if you don't use theme.
If use vendor:
buf generate with buf generate --exclude-path vendor.buf format -w with buf format -w --exclude-path vendor.If you use grpc-gateway:
import "third_party/googleapis/google/api/annotations.proto"; with import "google/api/annotations.proto";security_definitions, security, in option (grpc.gateway.protoc_gen_swagger.options.openapiv2_swagger).The last step is delete prototool.yaml.
If you are not migrate but start from scratch:
buf lint to make sure your proto is good.buf breaking --against "https://your-grpc-repo-goes-here.git" to make sure each time you update proto, you don't break backward compatibility.Run make gen to have fun of course.
Remember grpc-ecosystem/grpc-gateway, envoyproxy/protoc-gen-validate, kei2100/protoc-gen-marshal-zap is optional, so feel free to delete if you don't use theme.
If use vendor:
buf generate with buf generate --exclude-path vendor.buf format -w with buf format -w --exclude-path vendor.If you use grpc-gateway:
+import "third_party/googleapis/google/api/annotations.proto"; with import "google/api/annotations.proto";
+security_definitions, security, in option (grpc.gateway.protoc_gen_swagger.options.openapiv2_swagger).The last step is delete prototool.yaml.
If you are not migrate but start from scratch:
+buf lint to make sure your proto is good.buf breaking --against "https://your-grpc-repo-goes-here.git" to make sure each time you update proto, you don't break backward compatibility.There come a time when you need to experiment new things, new style, new approach. -So this post serves as it is named.
Instead of:
type Client interface {
- GetUser()
- AddUser()
- GetAccount()
- RemoveAccount()
+
+
+
+
+
+
+
+
+
+ Index
+ Experiment Go
+There come a time when you need to experiment new things, new style, new approach.
+So this post serves as it is named.
+Design API by trimming down the interface/struct or whatever
+Instead of:
+type Client interface {
+ GetUser()
+ AddUser()
+ GetAccount()
+ RemoveAccount()
}
-// c is Client
-c.GetUser()
-c.RemoveAccount()
Try:
type Client struct {
- User ClientUser
- Account ClientAccount
+// c is Client
+c.GetUser()
+c.RemoveAccount()
+Try:
+type Client struct {
+ User ClientUser
+ Account ClientAccount
}
-type ClientUser interface {
- Get()
- Add()
+type ClientUser interface {
+ Get()
+ Add()
}
-type ClientAccount interface {
- Get()
- Remove()
+type ClientAccount interface {
+ Get()
+ Remove()
}
-// c is Client
-c.User.Get()
-c.Account.Remove()
The difference is c.GetUser() -> c.User.Get().
For example we have client which connect to bank.
+// c is Client
+c.User.Get()
+c.Account.Remove()
The difference is c.GetUser() -> c.User.Get().
For example we have client which connect to bank.
There are many functions like GetUser, GetTransaction, VerifyAccount, ...
-So split big client to many children, each child handle single aspect, like user or transaction.
My concert is we replace an interface with a struct which contains multiple interfaces aka children. -I don't know if this is the right call.
This pattern is used by google/go-github.
Why? +So split big client to many children, each child handle single aspect, like user or transaction.
+My concert is we replace an interface with a struct which contains multiple interfaces aka children. +I don't know if this is the right call.
+This pattern is used by google/go-github.
+Why?
See for yourself.
-Also read A new Go API for Protocol Buffers to know why v1.20.0 is v2.
Currently there are some:
v1.20.0 is v2.
+Currently there are some:
+Previously in my fresher software developer time, I rarely write SQL, I always use ORM to wrap SQL. + + +
+ + + + + + + Index +Previously in my fresher software developer time, I rarely write SQL, I always use ORM to wrap SQL. But time past and too much abstraction bites me. So I decide to only write SQL from now as much as possible, no more ORM for me. -But if there is any cool ORM for Go, I guess I try.
This guide is not kind of guide which cover all cases. -Just my little tricks when I work with SQL.
Use UUID instead. -If you can, and you should, choose UUID type which can be sortable.
Stay away from all kind of database timestamp (MySQL timestmap, SQLite timestamp, ...) -Just use int64 then pass the timestamp in service layer not database layer.
Why? Because time and date and location are too much complex to handle. +But if there is any cool ORM for Go, I guess I try.
+This guide is not kind of guide which cover all cases. +Just my little tricks when I work with SQL.
+Use UUID instead. +If you can, and you should, choose UUID type which can be sortable.
+Stay away from all kind of database timestamp (MySQL timestmap, SQLite timestamp, ...) +Just use int64 then pass the timestamp in service layer not database layer.
+Why? Because time and date and location are too much complex to handle. In my business, I use timestamp in milliseconds. Then I save timestamp as int64 value to database. Each time I get timestamp from database, I parse to time struct in Go with location or format I want. -No more hassle!
It looks like this:
[Business] time, data -> convert to unix timestamp milliseconds -> [Database] int64You should use index for faster query, but not too much. +No more hassle!
+It looks like this:
+[Business] time, data -> convert to unix timestamp milliseconds -> [Database] int64You should use index for faster query, but not too much. Don't create index for every fields in table. -Choose wisely!
For example, create index in MySQL:
CREATE INDEX `idx_timestamp`
- ON `user_upload` (`timestamp`);If compare with field which can be NULL, remember to check NULL for safety.
-- field_something can be NULL
+Choose wisely!
+For example, create index in MySQL:
+CREATE INDEX `idx_timestamp`
+ ON `user_upload` (`timestamp`);
+Be careful with NULL
+If compare with field which can be NULL, remember to check NULL for safety.
+-- field_something can be NULL
--- Bad
-SELECT *
-FROM table
-WHERE field_something != 1
+-- Bad
+SELECT *
+FROM table
+WHERE field_something != 1
--- Good
-SELECT *
-FROM table
-WHERE (field_something IS NULL OR field_something != 1)
Need clarify why this happpen? Idk :(
-VARCHAR or TEXT
Prefer VARCHAR if you need to query and of course use index, and make sure size of value will never hit the limit.
-Prefer TEXT if you don't care, just want to store something.
Be super careful when migrate, update database on production and online!!!
Plase read docs about online ddl operations before do anything online (keep database running the same time update it, for example create index, ...)
Need clarify why this happpen? Idk :(
+VARCHAR or TEXT
+Prefer VARCHAR if you need to query and of course use index, and make sure size of value will never hit the limit.
+Prefer TEXT if you don't care, just want to store something.
Plase read docs about online ddl operations before do anything online (keep database running the same time update it, for example create index, ...)
+My quick check for .gitignore.
# macOS
+
+
+
+
+
+
+
+
+
+ Index
+ gitignore
+My quick check for .gitignore.
+Base
+# macOS
.DS_Store
-# Windows
-*.exe
+# Windows
+*.exe
-# IntelliJ
+# IntelliJ
.idea/
-# VSCode
-.vscode/
Go
# Go
-# Test coverage
+# VSCode
+.vscode/
+Go
+# Go
+# Test coverage
coverage.out
-# Should ignore vendor
-vendor
Python
venv
Feel free to ask me via email
-Mastodon
\ No newline at end of file
+# Should ignore vendor
+vendorvenvThis serves as design draft of reload config system
@startuml Reload config
+
+
+
+
+
+
+
+
+
+ Index
+ Reload config
+This serves as design draft of reload config system
+@startuml Reload config
-participant admin
-participant other_service
-participant config_service
-participant storage
+skinparam defaultFontName Iosevka Term SS08
-== Admin handle ==
+participant admin
+participant other_service
+participant config_service
+participant storage
-admin -> config_service: set/update/delete config
+== Admin handle ==
-config_service -> storage: set/update/delete config
+admin -> config_service: set/update/delete config
-== Other service handle ==
+config_service -> storage: set/update/delete config
-other_service -> other_service: init service
+== Other service handle ==
-activate other_service
+other_service -> other_service: init service
-other_service -> storage: make connection
+activate other_service
-loop
+other_service -> storage: make connection
- other_service -> storage: listen on config change
+loop
- other_service -> other_service: save config to memory
+ other_service -> storage: listen on config change
-end
+ other_service -> other_service: save config to memory
-deactivate other_service
+end
-other_service -> other_service: do business
+deactivate other_service
-activate other_service
+other_service -> other_service: do business
-other_service -> other_service: get config
+activate other_service
-other_service -> other_service: do other business
+other_service -> other_service: get config
-deactivate other_service
+other_service -> other_service: do other business
-@enduml
Config storage can be any key value storage or database like etcd, Consul, mySQL, ...
If storage is key value storage, maybe there is API to listen on config change.
-Otherwise we should create a loop to get all config from storage for some interval, for example each 5 minute.
Each other_service need to get config from its memory, not hit storage.
-So there is some delay between upstream config (config in storage) and downstream config (config in other_service), but maybe we can forgive that delay (???).
Pros:
Config can be dynamic, service does not need to restart to apply new config.
Each service only keep 1 connection to storage to listen to config change, not hit storage for each request.
Cons:
storage.storage failure.Config storage can be any key value storage or database like etcd, Consul, mySQL, ...
+If storage is key value storage, maybe there is API to listen on config change. +Otherwise we should create a loop to get all config from storage for some interval, for example each 5 minute.
+Each other_service need to get config from its memory, not hit storage.
+So there is some delay between upstream config (config in storage) and downstream config (config in other_service), but maybe we can forgive that delay (???).
Pros:
+Config can be dynamic, service does not need to restart to apply new config.
+Each service only keep 1 connection to storage to listen to config change, not hit storage for each request.
Cons:
+storage.storage failure.Install Arch Linux is thing I always want to do for my laptop/PC since I had my laptop in ninth grade.
This is not a guide for everyone, this is just save for myself in a future and for anyone who want to walk in my shoes.
Check disks carefully:
lsblkCheck UEFI mode:
ls /sys/firmware/efi/efivarsFor wifi, use iwd.
cgdisk /dev/sdxUEFI/GPT layout:
| Mount point | Partition | Partition type | Suggested size |
|---|---|---|---|
/mnt/efi | /dev/efi_system_partition | EFI System Partition | 512 MiB |
/mnt/boot | /dev/extended_boot_loader_partition | Extended Boot Loader Partition | 1 GiB |
/mnt | /dev/root_partition | Root Partition |
BIOS/GPT layout:
| Mount point | Partition | Partition type | Suggested size |
|---|---|---|---|
| BIOS boot partition | 1 MiB | ||
/mnt | /dev/root_partition | Root Partition |
LVM:
# Create physical volumes
+
+
+
+
+
+
+
+
+
+ Index
+ Install Arch Linux
+Install Arch Linux is thing I always want to do for my laptop/PC since I had my laptop in ninth grade.
+This is not a guide for everyone, this is just save for myself in a future and for anyone who want to walk in my shoes.
+Installation guide
+Pre-installation
+Check disks carefully:
+lsblk
+
+Verify the boot mode
+Check UEFI mode:
+ls /sys/firmware/efi/efivars
+Connect to the internet
+For wifi, use iwd.
+Partition the disks
+
+cgdisk /dev/sdx
+
+UEFI/GPT layout:
+| Mount point | +Partition | +Partition type | +Suggested size | +
|---|---|---|---|
/mnt/efi |
+/dev/efi_system_partition |
+EFI System Partition | +512 MiB | +
/mnt/boot |
+/dev/extended_boot_loader_partition |
+Extended Boot Loader Partition | +1 GiB | +
/mnt |
+/dev/root_partition |
+Root Partition | ++ |
BIOS/GPT layout:
+| Mount point | +Partition | +Partition type | +Suggested size | +
|---|---|---|---|
| + | + | BIOS boot partition | +1 MiB | +
/mnt |
+/dev/root_partition |
+Root Partition | ++ |
LVM:
+# Create physical volumes
pvcreate /dev/sdaX
-# Create volume groups
+# Create volume groups
vgcreate RootGroup /dev/sdaX /dev/sdaY
-# Create logical volumes
-lvcreate -l +100%FREE RootGroup -n rootvolFormat:
# efi
+# Create logical volumes
+lvcreate -l +100%FREE RootGroup -n rootvolFormat:
+# efi
mkfs.fat -F32 /dev/efi_system_partition
-# boot
+# boot
mkfs.fat -F32 /dev/extended_boot_loader_partition
-# root
+# root
mkfs.ext4 -L ROOT /dev/root_partition
-# root with btrfs
+# root with btrfs
mkfs.btrfs -L ROOT /dev/root_partition
-# root on lvm
-mkfs.ext4 /dev/RootGroup/rootvolMount:
# root
+# root on lvm
+mkfs.ext4 /dev/RootGroup/rootvolMount:
+# root
mount /dev/root_partition /mnt
-# root with btrfs
+# root with btrfs
mount -o compress=zstd /dev/root_partition /mnt
-# root on lvm
+# root on lvm
mount /dev/RootGroup/rootvol /mnt
-# efi
+# efi
mount --mkdir /dev/efi_system_partition /mnt/efi
-# boot
-mount --mkdir /dev/extended_boot_loader_partition /mnt/bootpacstrap -K /mnt base linux linux-firmware
+# boot
+mount --mkdir /dev/extended_boot_loader_partition /mnt/bootpacstrap -K /mnt base linux linux-firmware
-# AMD
+# AMD
pacstrap -K /mnt amd-ucode
-# Intel
+# Intel
pacstrap -K /mnt intel-ucode
-# Btrfs
+# Btrfs
pacstrap -K /mnt btrfs-progs
-# LVM
+# LVM
pacstrap -K /mnt lvm2
-# Text editor
-pacstrap -K /mnt neovimgenfstab -U /mnt >> /mnt/etc/fstabarch-chroot /mntln -sf /usr/share/zoneinfo/Region/City /etc/localtime
+# Text editor
+pacstrap -K /mnt neovimgenfstab -U /mnt >> /mnt/etc/fstabarch-chroot /mntln -sf /usr/share/zoneinfo/Region/City /etc/localtime
-hwclock --systohcEdit /etc/locale.gen:
# Uncomment en_US.UTF-8 UTF-8Generate locales:
locale-genEdit /etc/locale.conf:
LANG=en_US.UTF-8Edit /etc/hostname:
myhostnameEdit /etc/mkinitcpio.conf:
# LVM
-# https://wiki.archlinux.org/title/Install_Arch_Linux_on_LVM#Adding_mkinitcpio_hooks
+hwclock --systohcEdit /etc/locale.gen:
# Uncomment en_US.UTF-8 UTF-8Generate locales:
+locale-genEdit /etc/locale.conf:
LANG=en_US.UTF-8Edit /etc/hostname:
myhostnameEdit /etc/mkinitcpio.conf:
# LVM
+# https://wiki.archlinux.org/title/Install_Arch_Linux_on_LVM#Adding_mkinitcpio_hooks
HOOKS=(base udev ... block lvm2 filesystems)
-# https://wiki.archlinux.org/title/mkinitcpio#Common_hooks
-# Replace udev with systemdmkinitcpio -Ppasswd# NetworkManager
+# https://wiki.archlinux.org/title/mkinitcpio#Common_hooks
+# Replace udev with systemdmkinitcpio -Ppasswd# NetworkManager
pacman -Syu networkmanager
-systemctl enable NetworkManager.service
+systemctl enable NetworkManager.service
-# Bluetooth
+# Bluetooth
pacman -Syu bluez
-systemctl enable bluetooth.service
+systemctl enable bluetooth.service
-# Clock
-timedatectl set-ntp trueAlways remember to check dependencies when install packages.
Sudo:
pacman -Syu sudo
+# Clock
+timedatectl set-ntp trueAlways remember to check dependencies when install packages.
+Sudo:
+pacman -Syu sudo
EDITOR=nvim visudo
-# Uncomment group wheel
+# Uncomment group wheel
-# Add user if don't want to use systemd-homed
-useradd -m -G wheel -c "The Joker" joker
+# Add user if don't want to use systemd-homed
+useradd -m -G wheel -c "The Joker" joker
-# Or using zsh
-useradd -m -G wheel -s /usr/bin/zsh -c "The Joker" joker
+# Or using zsh
+useradd -m -G wheel -s /usr/bin/zsh -c "The Joker" joker
-# Set password
-passwd jokersystemctl enable systemd-homed.service
+# Set password
+passwd jokersystemctl enable systemd-homed.service
-homectl create joker --real-name="The Joker" --member-of=wheel
+homectl create joker --real-name="The Joker" --member-of=wheel
-# Using zsh
-homectl update joker --shell=/usr/bin/zshNote: +# Using zsh +homectl update joker --shell=/usr/bin/zsh
Note:
Can not run homectl when install Arch Linux.
-Should run on the first boot.
Install Xorg:
pacman -Syu xorg-serverpacman -Syu gnome-shell \
+Should run on the first boot.
+Desktop Environment
+Install Xorg:
+pacman -Syu xorg-server
+GNOME
+pacman -Syu gnome-shell \
gnome-control-center gnome-system-monitor \
gnome-tweaks gnome-backgrounds gnome-screenshot gnome-keyring gnome-logs \
gnome-console gnome-text-editor \
nautilus xdg-user-dirs-gtk file-roller evince eog
-# Login manager
+# Login manager
pacman -Syu gdm
-systemctl enable gdm.service
KDE (WIP)
pacman -Syu plasma-meta \
+systemctl enable gdm.service
+KDE (WIP)
+pacman -Syu plasma-meta \
kde-system-meta
-# Login manager
+# Login manager
pacman -Syu sddm
-systemctl enable sddm.service
List of applications
pacman
Uncomment in /etc/pacman.conf:
# Misc options
+systemctl enable sddm.service
+List of applications
+pacman
+Uncomment in /etc/pacman.conf:
+# Misc options
Color
-ParallelDownloadsPipewire (WIP)
pacman -Syu pipewire wireplumber \
+ParallelDownloads
+Pipewire (WIP)
+pacman -Syu pipewire wireplumber \
pipewire-alsa pipewire-pulse \
- gst-plugin-pipewire pipewire-v4l2
Flatpak (WIP)
pacman -Syu flatpak
Improving performance
https://wiki.archlinux.org/index.php/swap#Swap_file
https://wiki.archlinux.org/index.php/swap#Swappiness
https://wiki.archlinux.org/index.php/Systemd/Journal#Journal_size_limit
https://wiki.archlinux.org/index.php/Core_dump#Disabling_automatic_core_dumps
https://wiki.archlinux.org/index.php/Solid_state_drive#Periodic_TRIM
https://wiki.archlinux.org/index.php/Silent_boot
https://wiki.archlinux.org/title/Improving_performance#Watchdogs
https://wiki.archlinux.org/title/PRIME
In the end
This guide is updated regularly I promise.
Feel free to ask me via email
-Mastodon
\ No newline at end of file
+ gst-plugin-pipewire pipewire-v4l2pacman -Syu flatpakhttps://wiki.archlinux.org/index.php/swap#Swap_file
+https://wiki.archlinux.org/index.php/swap#Swappiness
+https://wiki.archlinux.org/index.php/Systemd/Journal#Journal_size_limit
+https://wiki.archlinux.org/index.php/Core_dump#Disabling_automatic_core_dumps
+https://wiki.archlinux.org/index.php/Solid_state_drive#Periodic_TRIM
+https://wiki.archlinux.org/index.php/Silent_boot
+https://wiki.archlinux.org/title/Improving_performance#Watchdogs
+https://wiki.archlinux.org/title/PRIME
+This guide is updated regularly I promise.
+ + Feel free to ask me via email + Mastodon + + diff --git a/docs/2022-12-25-go-buf.html b/docs/2022-12-25-go-buf.html index aa5149b..a2fe95d 100644 --- a/docs/2022-12-25-go-buf.html +++ b/docs/2022-12-25-go-buf.html @@ -1,4 +1,39 @@ -IndexThere are 2 questions here. + + +
+ + + + + + + Index +There are 2 questions here. What is Buf? -And why is Buf?
Feel free to ask me via email -Mastodon \ No newline at end of file +And why is Buf? + + Feel free to ask me via email + Mastodon + + diff --git a/docs/2022-12-25-go-test-asap.html b/docs/2022-12-25-go-test-asap.html index 8f17a3a..62983cf 100644 --- a/docs/2022-12-25-go-test-asap.html +++ b/docs/2022-12-25-go-test-asap.html @@ -1,203 +1,268 @@ -IndexImagine your project currently have 0% unit test code coverage. + + +
+ + + + + + + Index +Imagine your project currently have 0% unit test code coverage. And your boss keep pushing it to 80% or even 90%? What do you do? -Give up?
What if I tell you there is a way? -Not entirely cheating but ... you know, there is always trade off.
If your purpose is to test carefully all path, check if all return is correctly. +Give up?
+What if I tell you there is a way? +Not entirely cheating but ... you know, there is always trade off.
+If your purpose is to test carefully all path, check if all return is correctly. Sadly this post is not for you, I guess. -If you only want good number on test coverage, with minimum effort as possible, I hope this will show you some idea you can use :)
In my opinion, unit test is not that important (like must must have). +If you only want good number on test coverage, with minimum effort as possible, I hope this will show you some idea you can use :)
+In my opinion, unit test is not that important (like must must have). It's just make sure your code is running excatly as you intent it to be. -If you don't think about edge case before, unit test won't help you.
When I learn programming, I encounter very interesting idea, which become mainly my mindset when I dev later. +If you don't think about edge case before, unit test won't help you.
+When I learn programming, I encounter very interesting idea, which become mainly my mindset when I dev later. I don't recall it clearly, kinda like: "Don't just fix bugs, rewrite it so that kind of bugs will not appear again". So in our context, there is some thing we hardly or can not write test in Go. -My suggestion is don't use that thing.
In my experience, I can list a few here:
viper.Get...). You can and you should init all config when project starts.Err...). You should move all global var to fields inside some struct.If you code Go long enough, you know table driven tests and how is that so useful. +My suggestion is don't use that thing.
+In my experience, I can list a few here:
+viper.Get...). You can and you should init all config when project starts.Err...). You should move all global var to fields inside some struct.If you code Go long enough, you know table driven tests and how is that so useful. You set up test data, then you test. -Somewhere in the future, you change the func, then you need to update test data, then you good!
In simple case, your func only have 2 or 3 inputs so table drive tests is still looking good. -But real world is ugly (maybe not, idk I'm just too young in this industry). Your func can have 5 or 10 inputs, also your func call many third party services.
Imagine having below func to upload image:
type service struct {
- db DB
- redis Redis
- minio MinIO
- logService LogService
- verifyService VerifyService
+Somewhere in the future, you change the func, then you need to update test data, then you good!
+In simple case, your func only have 2 or 3 inputs so table drive tests is still looking good.
+But real world is ugly (maybe not, idk I'm just too young in this industry). Your func can have 5 or 10 inputs, also your func call many third party services.
+Imagine having below func to upload image:
+type service struct {
+ db DB
+ redis Redis
+ minio MinIO
+ logService LogService
+ verifyService VerifyService
}
-func (s *service) Upload(ctx context.Context, req Request) error {
- // I simplify by omitting the response, only care error for now
- if err := s.verifyService.Verify(req); err != nil {
- return err
+func (s *service) Upload(ctx context.Context, req Request) error {
+ // I simplify by omitting the response, only care error for now
+ if err := s.verifyService.Verify(req); err != nil {
+ return err
}
- if err := s.minio.Put(req); err != nil {
- return err
+ if err := s.minio.Put(req); err != nil {
+ return err
}
- if err := s.redis.Set(req); err != nil {
- return err
+ if err := s.redis.Set(req); err != nil {
+ return err
}
- if err := s.db.Save(req); err != nil {
- return err
+ if err := s.db.Save(req); err != nil {
+ return err
}
- if err := s.logService.Save(req); err != nil {
- return err
+ if err := s.logService.Save(req); err != nil {
+ return err
}
- return nil
-}
With table driven test and thanks to stretchr/testify, I usually write like this:
type ServiceSuite struct {
- suite.Suite
+ return nil
+}
+With table driven test and thanks to stretchr/testify, I usually write like this:
+type ServiceSuite struct {
+ suite.Suite
- db DBMock
- redis RedisMock
- minio MinIOMock
- logService LogServiceMock
- verifyService VerifyServiceMock
+ db DBMock
+ redis RedisMock
+ minio MinIOMock
+ logService LogServiceMock
+ verifyService VerifyServiceMock
- s service
+ s service
}
-func (s *ServiceSuite) SetupTest() {
- // Init mock
- // Init service
+func (s *ServiceSuite) SetupTest() {
+ // Init mock
+ // Init service
}
-func (s *ServiceSuite) TestUpload() {
- tests := []struct{
- name string
- req Request
- verifyErr error
- minioErr error
- redisErr error
- dbErr error
- logErr error
- wantErr error
+func (s *ServiceSuite) TestUpload() {
+ tests := []struct{
+ name string
+ req Request
+ verifyErr error
+ minioErr error
+ redisErr error
+ dbErr error
+ logErr error
+ wantErr error
}{
{
- // Init test case
+ // Init test case
}
}
- for _, tc := range tests {
- s.Run(tc.name, func(){
- // Mock all error depends on test case
- if tc.verifyErr != nil {
- s.verifyService.MockVerify().Return(tc.verifyErr)
+ for _, tc := range tests {
+ s.Run(tc.name, func(){
+ // Mock all error depends on test case
+ if tc.verifyErr != nil {
+ s.verifyService.MockVerify().Return(tc.verifyErr)
}
- // ...
+ // ...
- gotErr := s.service.Upload(tc.req)
- s.Equal(wantErr, gotErr)
+ gotErr := s.service.Upload(tc.req)
+ s.Equal(wantErr, gotErr)
})
}
-}
Looks good right?
+}
Looks good right? Be careful with this. -It can go from 0 to 100 ugly real quick.
What if req is a struct with many fields? +It can go from 0 to 100 ugly real quick.
+What if req is a struct with many fields? So in each test case you need to set up req. They are almost the same, but with some error case you must alter req. It's easy to be init with wrong value here (typing maybe ?). -Also all req looks similiar, kinda duplicated.
tests := []struct{
- name string
- req Request
- verifyErr error
- minioErr error
- redisErr error
- dbErr error
- logErr error
- wantErr error
+Also all req looks similiar, kinda duplicated.
+tests := []struct{
+ name string
+ req Request
+ verifyErr error
+ minioErr error
+ redisErr error
+ dbErr error
+ logErr error
+ wantErr error
}{
{
- req: Request {
- a: "a",
- b: {
- c: "c",
- d: {
- "e": e
+ req: Request {
+ a: "a",
+ b: {
+ c: "c",
+ d: {
+ "e": e
}
}
}
- // Other fieles
+ // Other fieles
},
{
- req: Request {
- a: "a",
- b: {
- c: "c",
- d: {
- "e": e
+ req: Request {
+ a: "a",
+ b: {
+ c: "c",
+ d: {
+ "e": e
}
}
}
- // Other fieles
+ // Other fieles
},
{
- req: Request {
- a: "a",
- b: {
- c: "c",
- d: {
- "e": e
+ req: Request {
+ a: "a",
+ b: {
+ c: "c",
+ d: {
+ "e": e
}
}
}
- // Other fieles
+ // Other fieles
}
- }
What if dependencies of service keep growing?
-More mock error to test data of course.
tests := []struct{
- name string
- req Request
- verifyErr error
- minioErr error
- redisErr error
- dbErr error
- logErr error
- wantErr error
- // Murr error
- aErr error
- bErr error
- cErr error
- // ...
+ }
+What if dependencies of service keep growing?
+More mock error to test data of course.
+ tests := []struct{
+ name string
+ req Request
+ verifyErr error
+ minioErr error
+ redisErr error
+ dbErr error
+ logErr error
+ wantErr error
+ // Murr error
+ aErr error
+ bErr error
+ cErr error
+ // ...
}{
{
- // Init test case
+ // Init test case
}
- }
The test file keep growing longer and longer until I feel sick about it.
See tektoncd/pipeline unit test to get a feeling about this.
-When I see it, TestPodBuild has almost 2000 lines.
The solution I propose here is simple (absolutely not perfect, but good with my usecase) thanks to stretchr/testify.
+ }
The test file keep growing longer and longer until I feel sick about it.
+See tektoncd/pipeline unit test to get a feeling about this.
+When I see it, TestPodBuild has almost 2000 lines.
The solution I propose here is simple (absolutely not perfect, but good with my usecase) thanks to stretchr/testify. I init all default action on success case. Then I alter request or mock error for unit test to hit on other case. -Remember if unit test is hit, code coverate is surely increaesed, and that my goal.
// Init ServiceSuite as above
+Remember if unit test is hit, code coverate is surely increaesed, and that my goal.
+// Init ServiceSuite as above
-func (s *ServiceSuite) TestUpload() {
- // Init success request
- req := Request{
- // ...
+func (s *ServiceSuite) TestUpload() {
+ // Init success request
+ req := Request{
+ // ...
}
- // Init success action
- s.verifyService.MockVerify().Return(nil)
- // ...
+ // Init success action
+ s.verifyService.MockVerify().Return(nil)
+ // ...
- gotErr := s.service.Upload(tc.req)
- s.NoError(gotErr)
+ gotErr := s.service.Upload(tc.req)
+ s.NoError(gotErr)
- s.Run("failed", func(){
- // Alter failed request from default
- req := Request{
- // ...
+ s.Run("failed", func(){
+ // Alter failed request from default
+ req := Request{
+ // ...
}
- gotErr := s.service.Upload(tc.req)
- s.Error(gotErr)
+ gotErr := s.service.Upload(tc.req)
+ s.Error(gotErr)
})
- s.Run("another failed", func(){
- // Alter verify return
- s.verifyService.MockVerify().Return(someErr)
+ s.Run("another failed", func(){
+ // Alter verify return
+ s.verifyService.MockVerify().Return(someErr)
- gotErr := s.service.Upload(tc.req)
- s.Error(gotErr)
+ gotErr := s.service.Upload(tc.req)
+ s.Error(gotErr)
})
- // ...
-}
If you think this is not quick enough, just ignore the response.
-You only need to check error or not if you want code coverage only.
So if request change fields or more dependencies, I need to update success case, and maybe add corresponding error case if need.
Same idea but still with table, you can find here Functional table-driven tests in Go - Fatih Arslan.
Feel free to ask me via email
-Mastodon
\ No newline at end of file
+ // ...
+}If you think this is not quick enough, just ignore the response. +You only need to check error or not if you want code coverage only.
+So if request change fields or more dependencies, I need to update success case, and maybe add corresponding error case if need.
+Same idea but still with table, you can find here Functional table-driven tests in Go - Fatih Arslan.
+ + Feel free to ask me via email + Mastodon + + diff --git a/docs/index.html b/docs/index.html index c8f34d1..fb3733d 100644 --- a/docs/index.html +++ b/docs/index.html @@ -1,2 +1,50 @@ -IndexThis is where I dump my thoughts.
This is where I dump my thoughts.
+