How I Built This.
Let's start with WHY I built this.
I have vast experience in C++, .NET/C#/WPF, SOA, iOS/XCode/Xamarin and various other technologies, but I haven't really delved into web development or cloud-hosted solutions. Maybe it's my gut reaction to the wild browser inconsistencies 15 years ago where supporting multiple browsers was a pain. I also have spent the last 5 years writing little code and mostly managing people and teams, and so I wanted to get back into the nitty-gritty of coding again.
I basically set out to learn modern full-stack technologies. Some things I wanted to focus on:
WebUI, specifically javascript/typescript & React
Backend services, specifically node and/or lambda
Cloud deployments, specifically AWS
NoSQL DBs, specifically DynamoDB
Process
I researched various technologies -- primarily their popularity, acceptance, support, and future longevity to settle on the list above
I obtained an AWS Certified Cloud Practitioner certification after wading through all of the interestingly-named AWS services. I also learned a few things along the way too.
I found a problem to solve that would be somewhat interesting and involve (or could involve, even if contrived), various technologies.
Problem
My wife was addicted to, and got me addicted to a really ridiculous game called Ball Sort. The concept is simple -- sort the balls so that all of the same color are in the same container, with the only rule being you can only move a ball on top of another ball of the same color or into an empty container. I got stuck on a couple of levels and figured I could write an algorithm to solve this for me. In early stages I envisioned:
a UI where the user could setup the problem level
in a really-advanced version, using ML, get the problem level from a screen shot or something.
a backend service that would solve the game for the user
a database that could save off solutions to reduce compute time
possibly SQS/SNS to perform long-running solutions and report back to the user/app asynchronously.
Identity/Accounts?
A way to publish this in the cloud for external users. Focus on cost.
Would like a CI/CD pipeline to manage this.
Of course, a git repo to keep track of my work (also, I tend to switch back-and-forth between macOS and Windows and a git repo supports that habit, beyond simply being the "right" way to do development)
What I decidedly did not focus on:
Slick UI. Tickling pixels and tweaking colors drives me crazy; I decided I would be OK with focusing on functionality over prettiness. I do believe in great UIs, but that is not my focus area for this project.
Architecture
Things that I tried that did not work for me, or Things I Learned™:
For some reason I though hosting the web app on lambda would be smart (I think I heard that Pluralsight is deployed that way). The easiest way ended up being hosting the app as a static website on S3.
Sometimes things ARE setup right in AWS, but it takes a cache to expire, or things to propagate, or otherwise clean themselves up before they will actually work -- and many times I didn't have the tools or insight into what the delay was.
Calling lambda function directly from web app is a bad idea. It's hard, fragile, and forces you to publish secrets you shouldn't. Use API Gateway as an intermediary.
React Function components are the future; don't use class components
Lambda functions get initialized "once" and can serve multiple requests. This means you still need to be very careful with how things are initialized, cleaned up, and otherwise be careful of not having any secrets in memory that could be dangerous to have between runs. In my example, I had a "done" signal that wasn't being reset between "runs" so any subsequent run wouldn't do anything.
DynamoDB is fairly easy to work with. Hardest thing is formatting objects very verbosely, which AWS.DynamoDB.Converter helps a lot with. Use .input for attributes (fragments), use .marshall for full objects.
How to host a static website from S3, distributed by CloudFront to gain SSL (https) support, backed by an AWS Certificate and routed by Route53.
Predefined Games lambda was implemented originally as an as-simple-as-it-gets service to help me wire things up and learn that stack (basically "hello world", but reading contents from a file). The predefined games don't change that often, so I changed the app to load the predefined games from a json file on the web server (s3 static web site). This s3 file is updated as the records in the related dynamoDB table are updated. This is more compute-friendly, but the problem is that the CloudFront distribution does not react immediately to this change -- the default timeout is 24 hours. The current solution is to wait 24 hours for the file to propagate, or force a manual invalidation on the CloudFront distribution.
Versioning the s3 bucket was super easy and helped reduce time to update file on CloudFront. Also setup some lifecycle rules on this s3 bucket.
Other Considerations
Right now the application is designed and built for an audience of one - me. How would I approach this application differently if it were meant for other users, or a large user base? Here are some thoughts.
Scalability and Performance
Predefined Games. Two thoughts here...
If this were a super-large user base I would probably spend the time to define all of the games up-front before publishing the app. I may not even have a DynamoDB table to store this data and simply rely on the .json file. This way, I don't have to worry about adding games to the list as users create them, generating/updating the .json file, nor how browsers and CloudFront caches that file. Simplicity wins.
If I still needed a somewhat dynamic list of predefined games (e.g. user can define their own), I would probably look into a more direct way of querying the db -- bring back getPredefinedGames lambda but read from a db instead of a file, or use a graphql implementation to get the data, and throw some caching in front of it (since it rarely changes). Right now the .json file is essentially my poor-man's cache.
SolveGame
Improve algorithm. It is OK right now, but can take a few ms to few minutes to solve some games. It is depth-first to find "a" solution faster without necessarily finding the best solution or going through all possible combinations. There are some heuristics to reduce combinations to check (e.g. don't move a token from a column where the bottom 3 tokens are all the same color because that doesn't get you any closer to a solution, or bail on a solution after (# of containers * 5) moves because it will lead to a not-great solution. there are probably other algorithm optimizations that will help efficiency and performance.
Parallelize the processing (support horizontal scaling). Execution time could be reduced dramatically if I would spawn off additional instances/processes/threads and take a multi-pronged approach.
If the runtime cannot be reduced to something more reasonable (like < 10 seconds), implement a queueing + notification approach (e.g. SQS, SNS).
Security
Implement User accounts and access. It would be nice to track usage by user (application insights and analytics), but also to control access (and therefore my expenses). Down side is that creating an account, or even using something like Cognito or other Identity providers could be a barrier to entry for some.
Overall, the application is secure. There are fairly tight IAM roles and policies, but this could probably use a review.
Reliability
Add monitoring (canary calls, CloudWatch, alarms).
Automate build & deployment processes (more) for faster and more reliable rollbacks or patch/update deployments.
I'm already utilizing S3 & CloudFront for website deployment. Consider using lambda at edge.
Usability and Accessibility
UI is horrible, clean up rebuild so that it makes obvious sense for most people
UI is horrible, add/ensure accessibility components
I have only really done UI verification on Desktop Chrome & Edge. I know mobile Chrome & Edge have rendering issues on the thumbnail views.
TODOs / Next Steps
Solving games is a very expensive (compute, memory, time) operation. Work on queueing this work and having SNS respond when it is done.
UI is horrendous; needs lots of love
mouse-over for predefined games
predefined game selection -- highlight/bold or somehow indicate the game selected.
"reset"
thumbnails for each move / way to cycle through the solution moves / don't present 0-based indexes to users.
general layout and styling
GameSolution component... better state indication. Blank if nothing has happened, spinner if we are waiting on the response, clear/fade if the game or board changes
Simplify how data is structured and handled -- make it consistent
Add and re-instate tests that I lost when re-arranging code
Convert backend services to typescript?
general code cleanup and re-organization
CI/CD Pipeline (CodeDeploy?)
Comments