2018 May - Projects Log

I decided to split out these projects logs into months to prevent them getting too big. My top priority this month is getting my Library Checkout Sheet app basically done and tested before the school year ends, and to continue to monitor Electron (especially since Node.js 10 now has N-API stabilized), and learn Redux.

2018 May 1, Tuesday

This evening, I explored the Google Sheets API, V4, and successfully used the spreadsheets.values/get endpoint to retrieve the values in the cells of a Sheets document that I created and made public through a link. I had to create an API key so that Google would know whose using its API's to get data, which would be important for them if someone started flooding their API with spam requests. I also discovered the official Google API Node.js client, which should save me from having to write my own interface to the REST API urls. It would be a good exercise, but I just want to get this app done. I have yet to confirm that this library works for React-Native, and if not, then it should be pretty easy to create a basic url wrapper. Might even do it anyway and save myself a library.


2018 May 2, Wednesday

More progress made on the library checkout app - I can download data from a Google Sheets document from my app! It's still unprocessed, but the data is available and ready to be processed, in a nice and simple format: a 2-dimensional array of the values of the cells. To avoid putting my Google API key and the spreadsheet ID in my project's Git history, I put them both into a secrets.json file that I can keep separate from my code. This app is still in prototype stage, otherwise I would look into persistent App settings for storing this data, and I would probably build a way to scan a QR code of the API key, and maybe a way to browse the user's Google Sheets documents online, but I digress.

As for size, figuring about 100 characters per check-out/in entry (including student name, book title/author, out/in date and notes), and about 500 checkouts from my Dad's library per year, I'm looking at about 50 KB for a year's worth of actual data, so I'm not concerned at all about downloading the sheets document whenever the app starts up. I'm still trying to come up with a good plan for working with the offline data. I'm thinking I'll just save an array of records into my app's runtime state, and create a set of functions to ask questions about that data, all immutable and functional-like. These would be questions like: "Has this book been returned yet?", "What books does this student still have checked out?", and so forth. I think Rick Hickey is right; information is simple, don't complicate it! Separating processing logic from the data itself is a good guideline for creating apps that are simple and easy to reason about.


2018 May 3, Thursday

Just a little bit of work on the library checkout app - at this point, I have saved the downloaded Google Sheets data as a processed array of simple objects on the main form component. I'll be able to use this data to actually handle the form data, so I consider one direction of the Google Sheets interaction basically done. Eventually, I'll save new data to the Google Sheets document after a book is checked in or out.

I also thought it might be a good time to try out Neovim's integration of the Language Server Protocol, one of the proportionally few things I give Microsoft positive credit for. It was a bit tough, climbing the learning curve of integrating the client and a JavaScript language server into Neovim, and it's still not ideal, but JavaScript-intelligent operations like "Go to Definition" and "Rename", and "Hover" do basically work, though I hope to do more refining of my user experience. The Language Server Protocol ecosystem is still pretty young, but I have hopes that it will improve, and bring a high base level of functionality to any editor that supports the protocol, which is great news to any fringe editors that don't have a lot of developer resources.


2018 May 4, Friday

See my post: Learning Redux, with some musings


2018 May 5, Saturday

I'm introducing myself to Electron more, mostly focused on learning the ecosystem of tools that people use. Most of my investigation is starting at the electron-react-boilerplate project, which comes with a pre-setup build, development, and testing system that comes pretty close to what I want to start with. Starting from Electron's page on Boilerplates and CLIs, I believe a boilerplate is more educational, not to mention flexible, compared with using a tool that hides the different parts of the process that I may wish to tailor to specific projects. Boilerplates also are closer than command line tools are to something that I would set up from scratch, as far as the packages that are involved go. It's not perfect for everything, but it seems like a good place to start learning.

I also did more work on the library checkout project. (I'm wondering if I should split this out into its own post.) I was tired of the camera always staying on and draining battery, so I made a useful feature to close the camera after 60 seconds of having scanned something, and replace it with a button to re-activate it. Now I can set my phone down, and pretty soon, the camera shuts off.


2018 May 7, Monday

Tonight, I made great progress on the Library Checkout App - I used the Google Books API to perform an ISBN lookup of author and book title. The flow is that the user scans a book, and about half a second later, the ISBN, Book Title, and Book Author are all filled in, having been asynchronously retrieved through JSON. I'm making sure to display informative errors if the ISBN was not found, or if a network error occurred. I love how in the middle of an async function, since Promises are the underlying mechanism, I can put a .catch(e => {doSomethingWithError(e)}) where I would just await the promise, and any errors will go through that block, and not cause any issues for the rest of my code! I really find it beautiful how results and errors are handled by Promises in JavaScript.

I found out, though I didn't implement it yet, that Google Sheets API is pretty convenient for updating data in a spreadsheet. It is very convenient that the API has simple commands to update a range in-place with new values, or append a new row to a range of data. I plan on having a couple pure functions to return a new record (which will now contain their 0-based row index) to update or append to the spreadsheet, depending if checking out or returning a book. Now, next time I work on it, I'll actually be using the records and doing interesting data-related things with them, and I can't wait!


2018 May 8, Tuesday

Well, I didn't quite get to the fun data processing I wanted to. Instead, I fleshed out the 4 Google Sheets API methods that I want to use: get(range), append(range, valueRange), update(range, valueRange), and clear(range), and the data layer that sits above the invocations to the fetch() API function. I am very thankful for the handy sidebar that Google provides to test out their APIs in their documentation pages, so that I could be sure of the exact format I had to send the request with. I would not have expected that I need to give valueInputOption to specify whether to auto-format dates and numbers when they get inserted into a spreadsheet, but without it, all I got was errors.

I also hoisted the records data, including future cached computations (such as the records that have not been checked back in, and the unique known student names) from out of the main form into the main app, since it's data that will be used everywhere, including the summary page. Without Redux, with state managed by React components only, state follows a simple rule - any shared state gets hoisted into parent components until there are no duplicates of it anywhere in the component hierarchy.


2018 May 9, Wednesday

So, I finally got around to implementing some processing functions to deal with the list of records. My current functions that I've made are getCheckedOutRecords(records: Record[]) -> Record[], getCheckedOutRecordsByStudent(records: Record[]) -> Object<string, Record[]>, and getStudentWithCheckedOutBook(records: Record[], book: string) -> string|null. These should be all pretty self-explanitory, and as a side note, I really like it when functions are so eloquently named and typed so that their behavior and usage is obvious from the name and type signature alone. I saw this in Ramda's documentation, with the groupBy function being an excellent example. (I should mention that I'm not using TypeScript for this project, with a few good reasons that I won't get into now.) For this project, I'm currently experimenting with Ramda to see if it's a good replacement for Lodash as a standard library of good JavaScript utility functions. I like what I see about Ramda so far, especially that its API design was built with heavy influence from Functional Programming practices, which Ramda's documentation pages can explain much better than me.

I came up with, and used, a useful tool for debugging state changes for React Components while I was extracting the main form into its own component. I was having an issue where I couldn't type in the text fields. A character would flash briefly, and then disappear. I knew that the state was getting saved poorly, but I didn't know where. In JavaScript, it is trivial to redefine a function to have special behavior. I wanted to see the arguments to all calls to this.setState(...) on the main Component, so at the bottom of the constructor() for the component, I added these lines:

    this.setState = (val, fn) => {
      console.log(`setState: ${JSON.stringify(val)}`);
      super.setState(val, fn);
    }

Piece of cake. I saw immediately what I was doing wrong when I tested it again, and soon fixed my careless oversight. For anyone curious, I was calling this.setState({fieldName, value}), when I should have been calling it like this.setState({[fieldName]: value}). It's a very handy JavaScript feature called Computed Property Names, but it doesn't work if you use it wrong!

I'm very close to actually getting the proof of concept done. All I really need now is a confirmation button that takes the form's values and call my Sheets API function for book checkouts or book returns. After that, I have a few good ideas for ease of use based on using information from the records list. A prototype should be ready soon, very soon...


2018 May 10, Thursday

Tonight, I made my Big Submit Button, and actually exercised all the little record processing functions I've made, save for actually submitting. I would have made unit tests, as pure functions are extremely testable, but I still haven't decided on a good way to do unit tests (though I have for integration tests) in JavaScript, though I'd prefer a style where the tests were as close to the function as possible.

Hot reloading does still seem a little bit unreliable. Sometimes it works really nice, and I see the change half a second later, other times, I need to reload the whole app; sometimes it reload the app unexpectedly, and sometimes, things get into a state that I have to restart the app from scratch on the Android end. Yet, I'm sure it will improve in the future, since even though React Native has been out since 2015, it technically hasn't passed version 1 yet, though it is probably the best platform for mobile apps right now. I wish it was a bit easier to debug the Flexbox layout to see why my button wasn't expanding edge-to-edge. Small complaints, and I'm sure I'll figure it out yet.


2018 May 12, Saturday

So tonight was supposed to be the final test, but though I sent the api request, performing a "POST" operation using an API key as a url param didn't seem to satisfy Google API's authorization requirements. I was pretty sure I had tested this before in the Google Docs helper, but I'm not sure how to mimic what it does using a simple REST query. Problems for later.

On the other hand, I discovered that apparently React Native has a particular issue with Flex Box how I'm using it. I was under the impression that "align-self" on a child element could override the parent's "align-items" property for an individual item - it would sure make sense, but it didn't seem to be working. For now, the styles were cleaned up, and it looks the way I want it, and I'm happy with how simple they are right now. As soon as I figure out how to properly use the Google APIs, every questionable step will have been complete.


2018 May 14, Monday

Work in progress of properly interacting with the Google APIs - see my post for details on how frustrating that was.


2018 May 15, Tuesday

Finally, success, and light at the end of the tunnel! So... I don't think I lost any hairs over this, though it is simply maddening trying to find the right Google Docs API page, especially when you know you've seen it before and can't find it back. I finally, using OAuth 2.0 for spreadsheet access, was able to append a new row to a spreadsheet using the filled out data in the app! At this point, there is a short checklist of things I need to do before releasing it for testing:

  1. Finish the Check-out experience (success message, clear form)
  2. Implement the Check-in/return path based on records data
  3. Add convenience buttons to pick a book that a student has checked out
  4. Add convenience buttons to pick the student that has the book checked out
  5. Release a standalone app on the App Store!

That blog post is still coming, and will likely be finished once I publish the standalone app, which will finally conclude my experience with Google APIs authentication.

EDIT: App's not finished, but the post/rant can be found here.


2018 May 16, Wednesday

At this point, Item 1 is done, along with not asking the user for their consent every time. My strategy is to save the accessToken, and when the app starts up, I basically test it, and if it fails, then I ask for the user's consent again. If that fails again, well, time to give up and ask for my debugging skills! For a prototype, I think this passes the threshold for usability. I think I'll add one more item to the list in place of #5:

  1. Implement summary page showing stats on currently checked out books
  2. Release a standalone app on the App Store

One down, Items 2-6 to go...


2018 May 21, Monday

After a short break, I got around to returning books - that is, I can now mark books as returned from the app, if I give the same book title, author, and student name. It's not the best user experience as it could be though, given all the information I have about these records already. So with step 2 done, I'm on to steps 3 and 4.


2018 May 22, Tuesday

Tonight, I did some actual "work" work from home, which was actually fun because I got to setup a remote-access workflow through SSH to my computer at work. One of the many reasons I love using Vim as my text editor/IDE is that it works very well though SSH, as do all things that run in the command line interface. With secure port forwarding from my work computer to my home computer through the same connection, I was able to work on and debug a web-based application, viewing it in a browser on my home laptop, while keeping the code and text editor all on my work machine, a few miles away. What can I say? The Internet is amazing, SSH is amazing, Vim is amazing, and I am... pleased things worked so well, of course!


2018 May 26, Saturday

I needed a break. I can't fill all my evenings with more technology, otherwise I think I really will go insane. I was rushing on the library checkout sheet app because I thought we would want to test it before the school year ends, but we figured out we can test it in the summertime with a smaller data set, so there's no need to rush. I will definitely finish this project though - all the hard parts are done, and the app is usable as it is, and there's only a few user experience improvements to be made yet before I want to get feedback on it.

Tonight, I'm studying Material UI, a project started in 2014 that made it to stable version 1.0 this week, that provides React components based on Google's Material Design to create intuitive and beautiful user interfaces. The main reason is that I need a replacement for Bootstrap 3, because Bootstrap 4 is stable, but the project went through a lot of changes, so it's time to evaluate user interface libraries once more. I've played around with Reactstrap, which provides React components for Bootstrap 4, and I do think it's better than React Bootstrap, which is what I'm using for Bootstrap 3 right now, but my preference right now is leaning toward Material UI. The things I like so far about it are that the CSS-in-JS styling approach in Material UI seems much more powerful than Bootstrap's traditional style sheets, plus Material UI seems to be much more thought out in its semantics of what a meaningful user interface should look like, and components are more self-supporting - one could use just the Button component as any other JavaScript project without any other hassle besides import in a React Component.

Bootstrap is the 2nd-most starred Github project for good reason - it is incredibly popular and widely used, and a good modern user interface framework, but I personally think it was mostly modern for its time. Though Bootstrap is still a good tool, Material UI is now very popular as well. I think it deserves a good look to see if it's better for a given project than Bootstrap. And yes, I believe that there are measurable things in the context of a company or project that make one framework or library better than another - things like final project size, how complicated code is for specific desired user interfaces, degrees of customizability (including runtime changes), documentation and resources for help, etc.. So far, it looks like a solid project with many months of work put into it, and I'm looking forward to learning more about how to use it.


2018 May 28, Monday

Tonight's small amount of work involved testing and succeeding in properly applying React Native's ScrollView component inside of a React Navigation's Tab navigation content screen, with padding around the outsides, behaving as I would expect. The key for me in terms of styling was flex: 1 on the outermost parent containing my root navigator, no styling on the ScrollView, and ScrollView's contentContainerStyle set to a padding of 15, and inside the ScrollView, just a bunch of regular components, laying out properly according to Flexbox rules. My aim with this is to be able to click the submit button with the keyboard open, and this exercise was educational, and should help me achieve that goal.


2018 May 29, Tuesday

As a practice and exercise to become intimately familiar with Material UI, I am going to be using it for one of my past projects that I want to get done within a year: a Genealogy web app. I was using Bootstrap 4 with Reactstrap, but wasn't liking that too much, so I'm going to use some of the direction provided by Material UI to make an intuitive editor that works well on mobile.

Since it's been a while, tonight's task was updating EVERYTHING - all npm packages, Webpack 4, Babel Preset Env, React Hot Loader, everything. All this is being used from Webpack Dev Middleware and Webpack Hot Middleware so that I can start the entire Node.js application all at once. To do this, I had to strictly go through instructions on each individual project for setting up all the following things (in addition to modifying my previous webpack config for Webpack 4):

I remember being extremely frazzled the last time I put this whole thing together from scratch, but even though I still grant that this project has a lot of gears, it doesn't seem as much of a mess as it did before. I feel like I understand what each of these projects is basically doing, especially after reading Webpack's Hot Module Replacement concept docs. Things seem to be working right now without too much configuration nonsense, and I can't wait to figure out... why I can't authenticate to my obviously running ArangoDB Docker instance...

(23:35 Update - Ok, database works now! Now for Material UI...)


2018 May 30, Wednesday

As planned, I started adding Material UI to my Genealogy project. I also ended up replacing font-awesome with Material UI Icons instead. Even before I knew Material UI Icons existed, I think that SVG-based icons are much more flexible and appropriate way to do icons then Font Awesome's web font approach. CSS Tricks has a comparison between the two approaches here. There were some differences is icons, but most of them are replacable.

I also realized that I needed to severely redesign the user interface, and I think I have a decent plan for getting started with an editable outline view. I look forward to implementing it soon!


2018 May 31, Thursday

For my last record of the month, I actually took a side turn and return to the Rust programming language. I am somewhat interested in embedded devices, and I'm looking into the quite-popular hyper crate, an HTTP server library. I got an example project compiling and running, and for compiled native code, I do think it is a much better language than C++. As I play around with routes and large file serving, I will be investigating minimizing the final executable size for the tight space constraints of small embedded systems.

Rust seems to have support for many different platforms, to varying degrees, owing to its use of the LLVM ecosystem. LLVM is sort of a middle ground between code and machine code - languages like C++, Rust, Swift, etc... can compile to LLVM, and the LLVM compiles to any of the hundreds of different kinds of CPU artitectures out there. This makes a newer language like Rust much more broadly adaptable than if a separate compiler had to be written for every CPU architecture out there. The Teensy board is ARM-based as far as I can tell, and would be an interesting embedded target for certain devices.

* I also got the Language Server Protocol working for Rust's language server inside Neovim' LSP client, based on a very helpful blog post. Now with my favorite text editor, I get code completion, function documentation, inline compiler errors and warnings, navigation to definition, and everything else one can expect from a modern IDE. I don't usually give Microsoft credit when it comes to software engineering, but the LSP project is something I definitely approve of, as it creates an easy target for a lot of diverse code editors out there to get highly advanced language integration. My only possible complaint is that the client wants to do pretty much everything, taking away from good projects like the ALE Linter for Vim, but perhaps the standardization is needed, and also, that issue is not necessary LSP's fault. Now I can code and learn in Rust, with more available right from my editor. I look forward to further learning this languages and some of its special use cases.