posts-go/posts/2023-03-05-incident.md

1.4 KiB

Incident come and go

This is collect of all incidents I created in the past :(

Service starts with empty port

Because all configs is read from file.

But the port config is empty -> So when service inits, it use that empty port somehow.

Solution: For some configs, make sure to failed first if it's empty.

Race condition of series of APIs

For example I have 2 APIs:

  • API upload: allow user to upload image
  • API submit: submit data to server

API upload is slow, it takes 10s to finish. API submit is fast, only takes 2s.

The problem is submit use data from upload too. When user calls API upload, image is stored in cache. When user calls API submit, it use whatever image is stored in cache.

It's when the fun begins.

Imagine user Trong already upload image. So he is ready to submit. But for the same time, he re-call API upload to upload another image too.

So if API upload is finished first, which is kinda impossible (U know upload file is not fast right?), everything right. But for most cases, API submit is finished first. It means Trong's data is submitted with the old image. Then API upload is finished, it will replace the old image with the new one.So the old one, aka image in submitted data, is gone.

Chaos right there!

Solution: Use a lock, if user enter API upload, lock it to prevent user call other APIs. Rememeber to unlock after finished