What is FindNUS?

FindNUS is a lost and found web application that my pal Jin Xuan and I built over summer 2022.

FindNUS was awarded the highest level of achivement, given to only the top 7% of teams (top 29 of 414 teams) for NUS School of Computing’s 2022 iteration of Orbital, an independent software development module.

We built FindNUS because we felt that the current lost and found website in NUS sucks and we could do better.

You can try FindNUS out here!

We wrote a truck load of documentation, including things like User Research and obscure bug quashing. If this interests you, please head on over to our official docusite.

If for some inexplicable reason you want to read the technical nitty gritty and thought processes behind the backend design, please visit this subpage of the docusite.

It makes no sense for me to re-write many of the technical stuff which I covered exhaustively in the docusite; so this writeup will be a light reflection about various software engineering things I thought of when building the RESTful Backend API for FindNUS.

Building the Backend

All the backend is, in essence, is a bunch of API endpoints exposed to the frontend website to talk to the databases and perform business logic. Here’s how I planned out the architecture:

Architecture

The architecture follows the microservices design pattern.

Tech Stack 📚:
Golang, Go-gonic, MongoDB, Heroku, ElasticSearch, RabbitMQ, Firebase, Docker, SMTP, Imgur API

We seperated the business logic into logical, decoupled components that are the ‘microservices’. So we have the HTTP routing, CRUD logic and NLP as seperate microservices.

Microservice Ws

In theory and practice, I found that this decoupling is very useful. When building a new featureset such as the Lookout microservice, it was nice that when I introduced it to the testing environment, if when it breaks, the other microservices still functioned normally. This made testing on the frontend side still possible.

Microservices Ls

Microservices are good and all with the whole spiel about scalablility and decoupling, but it has its own pains. One thing is that unlike the Monolith architecture, setting up all the moving parts for microservices is a huge pain and timesink 😵.

Many components require lots of time and attention to set up, such as the Message Broker and Dockerfile config for containerization and Unit Testing code. Sometimes I felt like I was spending more time wrangling with RabbitMQ & Docker than writing the spicy business logic.

But I feel that for a project that really needs scale and fault tolerance, the effort to set up pays off, especially when 24/7 uptime is required. Monoliths are more dangerous in that sense - if one component breaks, its likely that the entire house of cards will fall too.

The cost is also substantial 💸, at least on a Hobby scale (I dont want to pay money yet). For Heroku¹, I can only spin up 2 containers at a go. Anything more, I would have to pay. So making a truly decoupled system with many microservices is simply unfeasible on a hobby/broke college kid scale.

From this I can see why some Orbital projects gravitated towards serverless design patterns. Why spend money to hire a backend engineer when all of that is abstracted away with AWS Lambda or whatever serverless providers that costs barely any cent per API call² exist out there?

Regardless, building a proper backend was fun, insightful and scratched that dev itch I had ever since ending my backend stint at TOFFS.

Unit Testing

Unit testing is in theory simple - write a function that calls your function to be tested. Give it some data with expected outputs and test to see if the function explodes 💥 or returns something wrong.

Its simple until you actually need to mock data which surprisingly takes a lot of effort if you want to do it right and cover the various permuatations and edge case behaviour. Even worse are writing unit tests for functions tightly coupled to external services like Databases or 3rd party APIs.

Careful thought needs to be put into designing the tests to not pollute the databases/apis (even for staging environments, because things will break!). Not to mention the many testing frameworks that exist out there.

This is why I like Golang. Writing unit tests for it and integrating it into CICD tools is simple(r).

I’m sure that there exist some unit testing frameworks for other languages out there that make things seamless and efficient. But in particular, building Unit tests is a language feature for Go and is quite easy to pick up.

A very non-production ready trivial golang unit test example:

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// foo.go
func Add(x int, y int){
    return x+y;
}

// foo_test.go
func TestAdd(t *testing.T){
    result := Add(10, 5)
    if result != 15 {
        t.Log("Expected 15, got", result)
        t.Fail()
    }
}

To test the Add function, just

0

 $ go test foo.go foo_test.go

Clean. 🤌 Go will output the PASS/FAIL status for each test that was defined in the *_test.go files. It also integrates well with Github Actions which makes automated testing simple and dare I say, fun.

The one thing I learnt about Unit Testing is that its payoffs is unbelievably good. It probably saved hours of my time from reactively dealing with bugs. It’s a nice early warning system and makes you think carefully about how to design your functions detect errors and handle them. 10/10 would do it again.

API Documentation - A Necessary Chore

When embarking on this project, I was really interested in trying to get industry-standard level of endpoint API documentation for FindNUS’s backend. I settled on OpenAPI3 specs because its interoperable with vendors that can process a OpenAPI3 yaml spec file and give really nice documentation.

However, most of these vendors cost $$$ so I opted to parse the yaml specs with a open-source tool openapi3-generator to help generate nice, human-readable formatted markdown docs from the spec file. In case you’re wondering, the yaml/json formats are not particularly nice to for a human.

Imagine digging through 500+ lines of this:

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# One of many endpoint definitions using OpenAPI3 specs
paths:
  /item:
    post:
      description: |
                Add new Lost item to be put on Lookout on the database.
      parameters:
      - in: header
        name: Authorization
        description: Firebase ID token of user 
        required: true
        schema:
          type: string
          example: "Authorization: my-firebase-idToken"
      requestBody:
        description: Callback item payload
        content:
          'application/json':
            schema:
              $ref: "#/components/schemas/NewItem"
      responses:
        '200':
          description: Item registered into database
        '400':
          description: Rejected new item into database
        '401':
          description: Firebase credentials not invalid

Not exactly human friendly.

The purpose of API Docs is to make things crystal clear and save time communicating with others³. But this only works if docs are updated and written well. In FindNUS’s case, due to the tight timeline and speed of development, I often had to play catch-up for documentation⁴.

Throughout the development cycle, as I added in new backend endpoints & updates, bits and pieces of the API docs weren’t updated. This led to my frontend teammate often referring to the API docs and deferring back to me when he realised that the docs don’t match up actual backend behaviour.

This defeated the purpose of having API docs as a communication aide. We lost quite a few hours of time playing the telephone game due to outdated docs.

Ultimately, maintaining good docs requires a lot of discipline on the developers' side. It’s not necessarily a trivial affair. Despite the pain of maintaining docs, it is still a necessary chore. If done well, development can be fast and seamless (which did happen at the later stages of FindNUS' development. We pushed out quite a few new features in a short span with minimal telephoning).

All in all, playing the ☎ game back and forth for code is not fun and could be better spent doing more fun things, like breaking the server 😉.

Conclusion

This project was a lot more taxing than I anticipated - partly because I was juggling this project with my full time internship. But it was particularly rewarding because it was a great opportunity to actually develop from scratch a project with proper CI/CD, Unit Testing and structured documentation, which is a good exercise for projects in the ‘real’ world.

Last but not least, special thanks to my buddy Jin Xuan 💪, the frontend dev and fellow Computer Engineer for this project.

Thank you for tanking the Frontend, and being such a patient & understanding teammate to work with. It wasn’t easy juggling and accomodating each other’s hectic schedules, but I’m glad we managed to pull through with a polished product and exceeding our own expectations for this project. It was an absolute joy working with you - FindNUS would not have been possible without you 💯.