Google Photo – Advanced Reuploading

tl;dr this long post goes into specifics of (1) how to fully delete all photos from Google Photos storage, and (2) how to re-upload your photos to take advantage of Pixel free storage tier.

Read this if you

  • are an advanced user,
  • use Google Photos,
  • don’t want to pay for extra Google Storage,
  • want to download all your photos,
  • or want to delete all your photos.
  • and want to re-upload everything via a Google Pixel phone.


The simple way

If you don’t want to get your hands dirty or get into complicated settings, I strongly recommend you upgrade to a higher tier Google Drive storage. Go to https://drive.google.com/settings/storage and select the 100 Gb or 1 Tb storage options. They are very good and you get to upload a ton of files for backup, sharing and collaboration. I feel that Google storage tiers are way more worth it than iCloud or Dropbox storage. Please use this. Also, safe and secure backup for your most important documents is way better than what most home users are able to do at home.

Google Pixel’s free photo storage
So I have a new Google Pixel 2 phone, which is absolutely awesome and I would highly recommend to everybody. This phone comes with free unlimited Original quality Google Photos storage to encourage you to take advantage of the super awesome camera. Standard tier by Google only offers free High Quality storage which means the image file is compressed and resized by their special algorithm. If you are a photogeek and love your original files dearly, you would want to avoid this. Although I will also recommend turning on the standard High Quality storage for all your family members if they do not have any other storage or backup available. It is a lifesaver for people who lose their phones or break their phones. The only reason ever to disable this is if you know it is a totally disposable phone or they have bandwidth/wifi issues. Otherwise, backup always on.

Photos from other sources
One interesting thing about the Pixel free storage is that any photos/videos uploaded through the Pixel phone is considered as eligible. This means that photo doesn’t have to be taken by the Pixel. It can be via your DSLR/hand held camera/other phones/previous phones etc etc. What if you have a ton of photos uploaded previously via other phones or uploaded via the web? They will still count as paid storage, unless…

So my plan is to download all my photo/video files out of Google Photos, and then reupload them via my Pixel phone. Fairly simple in theory, but rather tedious by hand, with caveats. I can understand why Google wouldn’t want to make this too easy with just a click of a button, but I guess the extra work I put into it is a fair sacrifice for this policy loophole they have left in place. Thanks Google. And really, the free storage is worth a fair bit of money as you’ll never get this with Apple or Dropbox. Google Photos is fully backed up, redundant storage, global fast availability, plus AI/ML features.

I will now explain through the steps to take, and things to watch out for. Depending on the amount of files you have, your timing and effort will vary considerably.

Step 1. Check current storage:
Go to https://drive.google.com/settings/storage
Log in under your Google account and see how much Total Storage you used. If you have plenty of free space, you might want to skip this whole exercise. Again, this is an advanced class.

Click on View Details. You will see a breakdown of Drive, Gmail and Photos storage. That’s as much information as you’ll get. Drive storage is Google Drives files, you can manage that from drive.google.com, Gmail storage are your email files, you can trim your emails if need be. Finally Google Photos storage shows the number are we are looking for here. I started out with 130 Gigabytes used by Google Photos, which is way over the usual free 15 Gb for normal Google accounts.

(note that my final downloaded amount was higher than what was shown in storage used. I believe that’s due to extra files that were stored for free and didn’t count towards storage.)

Step 2. Download all your photos

We have to retrieve the original sized files, in order to re-upload them. There are some ways to do this.

Option A. Use Google Takeout 
https://takeout.google.com/?pli=1
This is a fairly convenient way that Google created to allow people to take out all their info if they want to leave Google services forever.

I had problems with it

 

  • Click “Select None”
  • Choose “Google Photos”
  • Choose “All Albums”
  • Go down to the bottom, click “Next”
  • Click through the options – you can usually leave it to defaults
    (If you know what you are doing, you might want to select larger file sizes if your computer can manage it, makes it a bit easier to work with. I had almost 20 x 10Gb files.)

Option B. Use Google Drive

Under Google Drive, you can select to enable to show the Photos shortcut. This enables you to see your Google Photos files inside Drive. It’s sorted by year, but not much more is visible. But it is sufficient to run downloads. 

I selected one year at a time, and ran the download. I actually found this method better than using Takeout as I get more manageable files for this purpose. For huge years, I selected 3 months at a time to make the files smaller. Note that it splits into 2gb files by default, you cannot change this.

I still kept both Takeout and Drive downloads as I was afraid of losing files. Make sure to have sufficient hard disk space to do all this. If you had to choose one, I would go with Drive download. Make sure all the years are showing up though.

Option C. Use web photos

You should be able to select entire months for download. I had 11 years worth to download, so I did not want to select 11*12 times, I prefer to use Option B.

Step 3. Delete all current files on Google Photos.

Option A. Delete via Drive

Some people suggest deleting it via Drive. That did not seem to work for me. While the folders seem to disappear, the photos still keep showing up both on photos.google.com and on my Android app. Not Recommended.

Option B. Delete on web photos.google.com

In the web, there’s no super shortcut to do this, but some people suggest click-n-drag to select all photos and delete. You can do this to a certain extent – probably to hundreds or something, but it wasn’t good enough for me. It could not handle 1000s or 5000s reliably. Not Recommended.

Option C. On web.

You can also select Full Days at a time. While browsing the photos, click on the date field and it will show a selector. This is a good way to delete Entire Days.

Option D. On Android App.

On the app, you can do a few things: select days, ‘click n drag’, and also select full months. Unfortunately you cannot select full years, but selecting full months works well for me. Caveat: if you have huge months with eg 2000 photos from a trip, it won’t seem to delete fully. This is either due to caching or trash size limit or cap of 500 photos per delete or something else.

To get around this, I find it more reliable to use option C’s deletion of entire days to be better for heavy days. Overall I used a mix of C and D to fully clear all my photos. Keep emptying the trash / recycle bin as well. It takes hours to do this for 11 years worth of photos. Oh Google why do you do this to people.

It’s kind of rare you would ever want to fully delete all your precious photos though.

Step 4. Check

After deleting all the photos you see, return to Step 1 and check that your storage usage has reduced – the photos portion at least. See if it is showing 0 Gb used.

For me, there were still remaining files. I left it overnight to see if the servers were still syncing, but it was still there next morning. I really do not see any more photos under web or Android app. I concluded that the remaining images must be from Google Maps Review contributions or Google Maps Local Guides contributions or Google StreetView contributions. Depending on how you do the upload, these might ‘cost’ you real storage. (If you want to get really into this, go to the Local Guides forums where they will explain to you in excruciating detail).

So what do I do? I didn’t want to remove those photos, even if I wanted, it’s too tedious. The easiest way to do this is to ask Google Photos to compress all your existing remaining media into free High Quality version. That’s plenty good enough for free contributions that I’m not paid for. To do this, go to web/android Google Photos and find the compress function. It will convert all original full sized images into free storage images. It took 2-3 hours for me, and it was a long time before my storage showed a decreasing amount. Check back after awhile and it should now show that Google Photos is using 0 Gb of storage. Note that this is a one-time thing, it should not affect future uploads. Future uploads are determined by the upload quality and you should use Original Quality for Pixel uploads. 

Step 5. Re-upload all your photos

Here’s another tedious bit with multiple steps. This takes a lot of time too.

5A. Expand/Extract/Unzip your downloaded files.

5B. Copy them onto your Pixel phone, into the DCIM folder. You can use Windows Explorer. (if you’re more advanced you can use ADB to do this. “adb push origin destination works” well)

You can probably copy them into other folders too, if you wish. I actually created DCIM\Upload in order to not confuse them with newer photos in DCIM\Camera

I recommend you copy like 10 to 20 Gb max at a time, let the phone digest it, upload it, then remove it and copy the next batch. Too many and Photos will strain. Try to limit it to 2000 items too. The upload speed isn’t terribly fast, I recommend leaving a batch of 2-3000 photos overnight to upload, on unlimited home wifi so you don’t get a hefty bill or drained battery.

5C. Let it scan

The reason is that when Google Photos see a huge new batch of files, it will immediately get to work by scanning all the new files, indexing it, checking which files are already backed up, preparing for upload, and then starting the upload. You can see this in the status bar on the top of the photo stream. It’s not as super charged as the web uploader, or the desktop backup uploader. After all, this is designed to run on the phone and not kill your mobile data or battery. So don’t overload it or it will likely hang. I tried a batch of 10,000 photos, with some duplicated folders, and maxed out my phone storage. It didn’t run. I had to delete and redo.

(Edit: I recommend batches of 1000 to 2000 photos at a time, which I feel is a reasonable amount for a Pixel 2 XL. I had success up to 4-6000 photos, but it took really really long to parse and upload (~12 hours or more), which was quite disturbing. )

If you’re unsure or stuck, you can Clear Data on the photos app. This helps a bit, but avoid doing this too much it takes a lot of time to re-cache.

5D. Wait till gphotos app shows backup completed

5E. Run Google Photos Android app > “Free up my storage”. This will help you to clear off files quickly.

5F. Checked for missed files.

I discovered there were some files left over after the “free up my storage”, and I wasn’t sure whether there were uploaded successfully. Simplest way for me is to move them into another separate folder, which triggers their auto upload again.

5G. Delete remaining files and folders as required.

5H. Repeat next batch

So the routine is:

Select batch 1 folders > Transfer to Pixel > Let it scan and upload > Wait until it shows Backup complete > Run Google Photos “Free up my storage” > and then check for files that did not upload. > Delete completed files or folder as necessary > repeat next batch.

Done

After this, you should have 0 Gb used in Photos, as long as all future images go through the Pixel phone. I expect this will take me 1 week to complete the whole exercise. Make sure you have a lot of unlimited high bandwidth wifi to do this.

Edit: It took me about 2 weeks from start to end, doing all the above steps in part each day. It might be faster for you if you didn’t have to redo some wrong steps like I did, or if your files are fewer. If your storage space is running out soon, you might want to start early. If your gdrive space is 100% filled, you might have difficulty receiving emails.

You’ll probably need to redo your albums, and sort out your new old files after all the re-uploading.  You will receive a huge amount of auto-awesome creations – do turn off gphotos notifications otherwise you’ll be spammed.

Extra tip, you can turn off wifi on your phone while you are copying files from PC to phone, that will stop gphotos from trying to upload immediately. I don’t like the phone processor and file storage trying to do two heavy tasks at the same time.

If you have any questions, you can find me on Twitter @jhender or Google+

Google Photo – Advanced Reuploading

Stonemouth

Stonemouth – Iain Banks

Scotland

The book is set in north Scotland; this is what I imagine of the landscape, a cold hazy little township with rolling greens, hills, etc etc. A corner of the country that kids grew up in, played together, fought together, drank together, and experienced so little yet so much. Slow quiet conversations, slow quiet sentences, some drunken revelry, some young relationships. ‘Tender’ was the word one of the reviewers used, and I gently agreed with that – Iain Banks (non-M) has this tender streak, yet mixed with a huge dose of illicitness and wildness, a type of raw wildness that you’d find in Scotland’s northern shores and weather. Haven’t read like this in awhile. Still prefer his Culture novels of course, but this is nice too.

/spoilers
Continue reading “Stonemouth”

Stonemouth

Kolkata

“How was it like going back to Kolkata?”, you ask. It was difficult to answer. To quote the book, I needed some time to think about how to express it linguistically, and also it’s complicated.

Firstly it was interesting bringing someone new along. Someone who has never been there, never travelled with us for work, and has probably heard some idea about Kolkata, and India, and my India self, but now needs to put it to test. It was more fun this way I guess, rather than going with other familiar jaded souls, but maybe it’s the same either way.

Secondly, it felt really familiar. Of course it would, this be after all my >50th visit, and so many many days that I could count and it would be tiring. The airport (albeit the newer airport) is familiar, the air, the weather, the drivers, the roads, the sights, the hotel, the rooms, the toiletries, the bed, the TV (changed slightly), the traffic, the buildings (changed slightly), the office (albeit the newer one), the lab, the people (albeit changed slightly), were all terribly familiar, in a good and bad way I guess.

It was good to be familiar. It felt like I belonged there, or maybe rather it felt like friends and family, rather than just formal work and formal touristy relations. It was good like that. Keeps you at peace, knowing what to expect, knowing how much you need to do and how much you can do and etc. It’s an incredible difference the second time you see something, more so when it is the 50th time and you are recognised by some. Maybe it was a little disappointing that not everyone recognised me, but then people change, maybe it was a little disappointing that some service standards are as high as before, but then corners need to be trimmed and costs reduced. Maybe I was way more frustrated with the internet than before, but I am so much used to 1 Gbps home fibre broadband and 4G mobile and faster phones than before.

On one hand it was fun to have somewhere to go to, but it also felt like this is so ephemeral and won’t be something that will be there when I grow old. After all these are all people in jobs and people move, after some time.

I loved meeting the hotel staff again, the waiters now restaurant managers, the concierge. There are probably many other guests that they know well, but it is nice to be one of them.

The roads are a little better paved now. Some other new highways are up. The driver took a slightly different route. A familiar junction sported a new flyover. I wouldn’t know the best route to direct a driver right now, whereas previously I was so familiar I could exactly state which road names and which way I preferred to take. Yeah.

A new Marriot hotel opened, a new outdoor theatre opened which looks super silly, some old markets are still exactly totally the same, malls the same, restaurant makeovers, new waiters who are incompetent, etc.

A few new eateries, too many same old places that I eat at.

Similar old flights.

Fairly fun work.

All in all, not too bad as work trips go.

Sometimes it also feels like I haven’t changed at all, and still doing the same thing travelling to the same place. 🙁
Flipping through the same TV channels (actually probably upgraded to better), and on the same crappy internet.

Kolkata

OLEDs

The recent furore over OLED display quality, only highlighted because of the rabid Google phone fans (and critics), have really brought OLED production into the news. It’s way more interesting than I imagined.

Interesting points:

  • Samsung Displays (the section of the mega Samsung responsible for manufacturing displays from phone sized to TV sized), makes most of the production-ready speed and quality of phone OLEDs in the world. >90% of the main market in some cases. Wow.
  • LG is like the second player, and LG’s tech – comprising both manufacturing capability, technical knowledge, quality control, is on the entirely up to 3 years behind Samsung. They are trying their best to catch up, and they are catching up, which will get better as they have more production units, but it is still expected to take 1-2 years worth. Even then, Samsung will probably still be the leader, but maybe not the whole dominating runaway leader by then, with their pricing reigned in somewhat.
  • The best Samsung displays – because in production you will have better manufactured parts in all variances, no doubt go to Samsung’s top end phones and possibly Apple’s phones since Apple will be paying top dollar and full premium for these exclusive reservations. Apple’s iPhone X uses Samsung OLEDs, especially the larger and higher res phone sized OLEDs, and these are still in relatively limited production quantity, hence the ipx is so low production, hence the ip8/8+ do not use OLEDs.
  • This is also probably the reason why Google’s Pixel 2 uses the smaller Samsung OLED which is produced in better and cheaper quantity, but did not manage to either secure enough supply of the larger screen or it is too expensive. The ipx is really expensive – a few hundred dollars more, and the S Note 8 is Samsung’s own phone so they can probably cost it in cheaper.
  • LG’s production of larger sized OLEDs is the alternative. With Google, and Apple, both making huge monetary support to LG to make sure LG can survive and invest into future production. Ie $1 billion support from these two companys each, that kind of support. But so currently LG has a tougher time making higher quality displays, and at a higher consistency level, but it is cheaper, it is available in bigger supply than the Samsung panels.
  • LG is bringing up more production factories, with newer tech, but it is taking time or slower supply or still in the pipeline, ie not ready for 2017 holiday season phones upfront.
  • OLED display production depends alot on a Canon tech called Canon Tokki, for making the wafers/chips, and this item is also in super limited quality – single digit production per year. Samsung has been buying this up non-stop, making it harder for LG and others to get hold of it.
  • There are other smaller players trying to get into OLED production but none of them are significant right now, especially in China.
  • More interestingly, both Samsung and LG have sufficient tech to produce the more fancy type of OLEDs like flexible displays, but consumer acceptance/demand might not be sufficient right now and production might not be ready. However it is in the future cards for sure. That’s going to be super interesting especially for wearables.
  • Apple iWatch uses LG OLED, LG V30 uses LG OLED.
  • TV sized OLED wise, LG is the runaway leader, with many other companies like Sony and Panasonic using LG made OLED in their TVs. Apparently the tech is quite different from phone to TV sized.
  • Last year’s Google Pixel had a severe shortage of displays, and maybe amongst other things.

Going by this, OLED will take over top and mid-to-high-tier phones by next year. Prices of the part should drop a bit if LG keeps up expansion. The tech should stabilize or get better. By 2019 it should be stable.

OLEDs

NYC

New York City, one of the last of the major cities that I had’t gone to. (Country capitals are not really counted because they are just boring administrative zones and cities of sheer population size doesn’t necessarily translate to interesting level.)

It’s so hard to find any photo that defines what NYC was like, it is so many things, and, the more you stay there, the more it is less about what you thought it would be.

So a typical google search for NYC would turn up the skyline, and the statue.

Lower Manhattan (click for larger)

We had our share of bright blue super sunny weather. In fact super hot. Getting around NY in tshirt and shorts just don’t feel right. Not quite a summer destination. Wow it was hot, for a Singaporean.

In a sense, these days with the Internet, photos, StreetView, everyone can see anything anytime, you don’t have to travel to know what something looks like. The point of going there is to live the city, ny style.

Which starts with their well-known, and currently half disastrous subway. It’s old, it’s noisy, cranky, confusing, doesn’t connect too well East to West, super lack of escalators and elevators, not the safest feeling with the homeless, and then there’s all the disruption and maintenance works.

But you get used to it, up the stairs, down the stairs, through the cranky dumb heavy turnstiles, magnetic swipe card, directions Uptown Downtown, nearest lines, exits NE SE NW SW. NY really lives and breathes by its subway, more so than many or almost any other city. The gridlocked roads and traffic are plain terrible, the parking is terrible (what kinda nonsense is $30/hr), the buses are terrible, hence the only reasonable way to travel around is via the subway. Which makes it wonderful, the city is built around and lives around the subway, you have dense neighbourhoods that are within close walking distance to multiple stations, you have lots of food, services, attractions, etc, all near stations. Even tourist attractions are easily accessible without requiring taxis. That’s in huge contrast to cities that don’t have an extensive subway, you’ll see huge distances to walk, huge parking areas etc. I love NYC for this. Same for Paris, London. Singapore still needs way more stops at closer intervals.

Next, the Times Square. I mean I knew we were going to stay near to it, I thought that was a superb idea in terms of convenience, I didn’t know what we were getting into. omfg we weren’t like in the middle, we were like one traffic junction away at between 7th and 8th Ave, but damn it was overcrowded with tourists, buses, traffic, homeless people, and probably the worst pithole city area of the entire USA. I don’t think any other city in US has this amount of density. So yes it’s a little too close, for most people. I live in a fairly crowded city, I am pretty comfortable with crowds, and I am fine with it, but I can imagine how this would be a disaster for most people I know. Also frankly Times Square is just an hotpot of subway lines, office buildings, same old big name mid market clothing stores, same name restaurants, and the slew of theatres that settled in. Basically it’s not like the peak of artform or anything.

Times Square

OLYMPUS DIGITAL CAMERA

Next up, theatres and Broadway. How nice of them to experience a surge in popularity in recent years, with soaring ticket prices, packed houses, and tons of shows opening. There’s really alot of shows, maybe more than London when I last was in London 10 years back. It’s crazy the crowds. Musicals are great, but it’s just one facet of life. Anyway I feel like there’s no real musical streets around here, it’s kinda dispersed and swallowed up by the Times Square kitsch crowd of tourists and blinky lights, so it doesn’t feel like the area is theatre-y or artsy in some form. Yeah they need to just close the streets to lousy traffic and have more of a feel. I think West End is a bit better in that way, not so stupid feeling. But there are more shows here.

NYC

NYC food

The food. The varied population gives rise to a huge range of tastes, from the american trash like sugar-laden donuts and iced coffee, to the fine dining, to the abundance of mexican, chinese, jewish, italian, etc. It’s great. It means you get to change around and eat different things everyday that isn’t terribly-ugly looking burgers and pancakes. Here are some pizzas, in the new york slice style. Big, flat, on a paper plate, hanging over the edge, hot and tasty. Eating this over the long run will no doubt make you fat and nutritionally unbalanced, as with most american cuisine, but it’s fun, and inexpensive relatively. I guess you could do dollar slices all day, but I suggest trying some others too. Joe’s Pizza here isn’t too bad. NY Pizza Suprema near to Penn Station is awesome as dine-in place, saw this featured in a YouTube video and it was a good recommendation. I really enjoyed both.

Bagel and cream cheese. Because I’ve never seen shops full of 50 different flavors of cream cheese, 10 different bagels, and much tastier than those ridiculous breakfast cafes. So yeah come have some. Brooklyn Bagel & Coffee is awesome, Pick-A-Bagel chain is more common, less awesome, but they have like hot breakfast stuff for cheap.

Food Truck

Food trucks. Really quite the NYC phenomenon, they are everywhere. Never really went after the famous ones. Nathan’s Hot Dogs are everywhere, I’m not impressed. Dinge’s Waffles have a few spots, but I didn’t want sweet nonsense. There’s a ton of the meat stuff like lamb on rice, etc etc. Seems good. Really they are just slightly more neat looking versions of street hawker stalls in Asian countries. A thing, but not a cuisine mind you. I think it touches their American dream thing – being able to rent/buy one on the cheap and just set up shop anywhere and make some money.

Churros Ice Cream Sandwich
Smorgasburg
Smorgasburg

Smorgasburg a weekend food market in hipster corner Williamsburg, Brooklyn. It’s so popular it’s apparently expanded onto Manhattan, which makes it so much more convenient because you don’t have to like trek all the way to this corner. Anyway it was wonderful. Very hipster, as you can see from my churros salted caramel ice cream sandwich with oreo powder. Yeah. Also lots of other fancy stuff that you don’t see in Asia, like Asian food made hipster, like African stuff, like ridiculous combinations.

Chelsea Market
Los Tacos

Chelsea Market and Los Tacos. Another hipster area. I don’t know whether it’s overly touristy, but the locals still like the place, so I guess kudos for staying in scope, same for Smorgasburg. I guess the real tourist are just stuck in Times Square. Or maybe the tour buses herd them outta this expensive city as soon as possible. So Chelsea Market, tacos and lobster are great, the rest are probably all good, but just not as superb value.

Katz’s
look at the sauerkraut
so old school

Finally Katz’s Delicatessen. I guess every single guide talks about this place, because it’s been around for so damn long. You can’t come here too often, you’ll die from the mountain of meat they try to serve you, but I really liked the Reuben Sandwich, and I really didn’t like the signature roast beef sandwich, because the roast beef is so weird. Hot dog was great, pickles were too huge, serving sizes were too huge, atmosphere was superbly comfortable and homey. Place was featured in the movie When Harry Met Sally, very very very good movie, go watch it. Anyway I would totally come back here a few more times, but I had too many things to try and too many places to go.

Shake Shack

Ok so you have to eat a burger I guess. You can go to Shake Shack, I love it, it’s great. It’s quite like er it’s just a fast food burger place? Yeah exactly, I don’t know why it takes a fast food burger place to give me a no-tipping required, decent service, pretty looking burger. Coming from Asia, burgers are neat and well prepared things when you eat out, even if you go to McDonald’s, Mos Burger, or a $20 cafe, the burger looks presentable. In US, so far I see alot of normal american cafes with burgers that are not presentable. How difficult is this? Apparently very, hence the locals like Shake Shack alot. Also apparently Five Guys, but I didn’t go because the exterior looked lame, but people keep praising it. If you can make an average idealistic image of a burger, you win, in the US.

One of my favorite meals and things to do, is to just sit in the park at lunch time and eat a sandwich or some takeaway food. I think that’s one of my favorite moments in European and western hemisphere cities. It’s just so nice, in the right weather. Go to Bryant Park, it’s super convenient outside a subway stop, it is surrounded by bakeries and shops, and you can buy from Whole Foods as well. It’s got the right amount of green space, park chairs etc. Union Square isn’t so good, and many of the other spaces are too small. Bryant Park is nice. Also beautiful NY Public Library in the back, go in and take a look.

Here’s hoping you get to enjoy your America trip and not hate the living lights out of that place, which isn’t difficult to end up doing.

NYC food

Google Cloud Architect

Received my Google GSuite Administrator and, more importantly, Google Cloud Architect certification in Aug. =)

Spent alot of time reading the docs and going through Coursera lessons, plus all the hands-on experience over the past 3/4 of a year really helped a lot.

Cloud Architect is geared toward use of Google Cloud Platform, focusing on almost all aspects of the networking, compute, machine learning, data management, identity, security, solutions design etc. It’s a super alot of information and products that are covered.

Gsuite is much simpler and covers setting up and managing a Gsuite domain, quite a lot of details but far less.

There were very few resources on Cloud Architect to go on. Here are some tips and areas to focus on:

  1. Make sure you have hands-on experience in setting up basic networks, VMs, resources. As much hands-on experience as possible. It will be super useful in having a feel of where things go and what workload or design is appropriate. You could do this before or after reading the docs and taking classes.
  2. Official docs at https://cloud.google.com/   Read through all the intros, docs, FAQs. There’s alot of good and updated information inside here. It’s quite lengthy and dry so you might want to do this back and forth with all other resources.
  3. Official study guide and case studies at https://cloud.google.com/certification/guides/cloud-architect/  The case studies are used in the actual exams and they might introduce more, who knows. So you should just read through and familiarize yourself beforehand as much as possible. Think about what you might design. You don’t need to memorize anything here.
  4. Coursera courses are recommended, and designed by Google official trainer. This set https://www.coursera.org/specializations/gcp-architecture is pretty good. You can just listen through the videos, try the guided tutorials which have good hands-on experience in most aspects. You can get through all four courses within a month very quickly. If you have extra time, you can also try the single course designed for AWS professionals, it has a slightly different perspective which is nice.
  5. I believe there is a new linuxacademy course but I didn’t try that.
  6. In-person courses weren’t really necessary, I didn’t have access and it was too expensive.
  7. Cloud Next Youtube videos are quite informative. But there are too many to finish. Worth a listen as they have nice real-life scenarios with real clients. Good tips in many areas.
  8. Knowing how to choose the appropriate storage tier, the appropriate monitoring tools, compute tools, networking tools, data management tools are totally essential and expected.
  9. Knowing how to debug certain errors or issues are good to know as well. Unfortunately this is hard to come by in lesson or tutorial which is why I strongly encourage hands-on experimentation or real-life use if possible. Although this might be tough to expect.
  10. Note that the exam was largely design at the start of 2017, same for the Courses, whereas online documentation is up-to-date and evolving every single week. GCP launches new products, new changes, updates, new price discounts like all the time. It’s a very fast paced platform, so some difference might be there. However don’t worry too much most of the concepts stay the same. Of course you can expect the newest latest tool will probably not be into the exam yet.

Good luck and have fun getting into GCP. It’s gonna be useful and informative even if you’ve a regular GCP user as the platform is so wide in scope.

 

 

GCP :

 

 

GSuite:

Google Cloud Architect

Ruby on Rails on Google App Engine

and the impossible number of problems that one runs into whenever trying to do anything code related.

Testing: Ruby on Rails 5.1.2 application to run on Google App Engine Flexible environment.

I wonder if running Ruby-2.4.1 on laptop vs the Ruby 2.3.3 on GAE will cause a problem….nvm.

Apparently GAE Flex can now run other languages by managing it inside a docker container behind that scenes. Well lo and behold, we would no longer need to set up Compute Engine, Container Engine and the likes. Let’s see how well this could possibly work as the community document is rather scarce, and the official docs are pretty lame by running a sample Rails app within gcloud cloud console instead of deploying a real-life app from desktop.

 

  1. First problem, production env key. Stupid Rails keeps doing this.
  2. GAE requires a /_ah/health check. Why can’t they be more upfront about this instead of failing stupidly in the background
  3. Realised the repeatedly deploying adds more versions into GAE, oh it doesn’t replace each other. I’m such a GAE noob.
  4. GAE version cannot be deleted unless it’s not serving. Oh. Then why make me wait so long to throw an error. Dumb GAE
  5. So many errors upon trying to stop/delete versions. Dumb GAE
  6. Need to add a controller and roots. Dumb me

Oh look it’s running finally

 

and also managed to route the domain too. Okay.

 

Ruby on Rails on Google App Engine

Change in the big world

EDITORIAL | MUSING

I wonder if people look at the world, feel that it’s too big, too problematic, too complicated, too difficult to change, and just decide to turn away / run away from it all.

When you realise that there are 6 billion+ living beings, and counting, and increasing faster than dying, when you realise that there are so many countries, so many ‘races’, so many religions, so many square kilometres, that you can’t even properly visit all the countries once in your lifetime, that you in your lifetime only know, interact and remember so few of them, when you realise how short your life and time is, that is when most people will just drop fancy ideas from their head and instead just grab that burger, take that commute home, watch some TV and settle in for yet another night, yet another weekend, yet another year end, yet another new year.

Indeed our little human lives are all too often so short. 24 Earth hours, 365.25 Earth days, 100 Sol years. And indeed most people do not end up making a huge change in the world during their lives, commonfolk aren’t heroes, generals, leaders, presidents, big bosses (of big companies), celebrities, etc. Heck even these people don’t always get to make significant long term lasting changes.

That’s probably why it’s so easy for us to turn away from it, to be sucked into being a part of the ecosystem, to run away into fantasy and movies and tv shows, and make-belief sports competitions, and self meditation.

But there’s always some people who feel that they could try to change something. Probably they never thought it would get so big, but it did, and it grew and grew. Change happens. Fortuitously, accidentally, decidedly?, endlessly.

The odd thing is that if you look back, you see all these massive changes happening, despite sometimes everything feeling that it might never change at all, being so set in stone and all.

The grand nations, the grand corporations, employing tens of thousands, hundreds of thousands of staff, seem like too big to fail, too old to fail, and to have been around ever all the while in our lives. But they weren’t, and they’ve come, gone, changed, disappeared, so many times within years or decades.

When I get old, perhaps I will look back to the time as having lived through the Age where there was widespread Internet version x, where Apple and Google grew big (might be bigger or smaller in the fuure), where SpaceX and Tesla appeared from nothing, where Netscape Navigator was a thing, where people first learned programming (not really it started before my time), and so forth. It never really felt like anything, but when we look back, it might just be. What did it feel like to live through the rock & roll? When Bob Dylan was alive? When x celebrity was alive? When JFK was President? When Trump was President? And all I could say was it just felt like the one life I was living, no different than any other possible life I was living, which is no other. It was as good as I could have, as bad as I could have.

I think it is good to have people who dare to change, to dream, and be different. Not to try to be different or special, but because they see that some things could be better. It would be a tad sad to just accept everything at face value as the way it will always be.

In a way it’s like a parallel of the chaos theory (which is frankly quite a simple theory but hey it opened people’s minds).

There was an article today about the definition of “Space” – what is space? Yes, mostly the outside of Earth, Space. It made for some interesting thoughts on how weirdly we define that word and how we understand it so counter-intuitively and in a conflicted manner. http://nautil.us/issue/49/the-absurd/what-is-space  The absurdity of it all, really. I’m surprised human brains don’t implode or remark at our own stupidity and narrow view of the universe/physics.

Which brings me to the supposed 4D / multi dimensional / multi whatever space of the Culture science fiction novel series by Ian M. Banks. The “Level 8” tech space ships travel and function in realms beyond our description and imagination, not that the book tries to dumb it down but rather knows it cannot explain it at all. As fictional as it is, it could well be an approximation of the truth. All this fancy futuristic stuff. I’m a bit sad I will not get to experience it. Cryo preservation is not ready, nor is it useful. Why in the world spend so much money, energy and resources to preserve someone when it is more useful to grow a new human who can learn new stuff and be more useful. It was worked for eons and might work for millenia to come.

Change in the big world

One weird trip to HK

It was quite an unconventional trip to HK.

nice hotel

Shenzhen

Rainy weather and sick person

Lunch with local friends

But I just can’t avoid good old Mongkok

Didn’t manage to take their iconic mini-buses this time actually

but I love the crowd

and many views from the streets, from above, from below

dang, using the pansonic 20mm f1.7 II does have quite a fascinating color saturation and perspective. cool.

One weird trip to HK