The funniest geotweeps

A week ago I asked Twitter to nominate candidates for “The Funniest Geotweep”. Here are the results, ranked high-to-low. I will keep the poll open a while longer to allow for late nominations, and will update the results periodically.

Poll results as of 2021-02-03T23:59:59Z:

Look, Ma! I made a mobile geodatabase!

I did, indeed. Prompted by a tweet from the ever-vigilant Randy Hale, I fired up my just-updated ArcGIS Pro 2.7, created a new mobile geodatabase, and exported a feature class into it.

First impressions: It works. It’s nice. It is new in ArcGIS Pro 2.7, and only works in ArcGIS Pro 2.7. ArcMap 10.8.1 doesn’t see it. It has many advantages (compact, modern, high size and name limits) over other geodata formats such as shapefile and file geodatabase. It is very similar to the geopackage in that it uses the open-source SQLite database. It also looks like it’s Esri’s attempt to push users off ArcMap and onto Pro by recreating the personal geodatabase for the 2020s. 

If you are a member of a GIS team where everybody uses only ArcGIS Pro 2.7 (and whose data partners also only use ArcGIS Pro 2.7), the mobile geodatabase looks like an attractive proposition. In my own work experience however (1, 2), no GIS data-sharing partner has ever requested or delivered anything but the much-maligned shapefile. YMMV, and this will likely change going forward, but as of today this is my reality.

So where does that leave me? What should I do with the mobile geodatabase today? I will dive deeper and try to discover all the advantages (and eventual drawbacks) of the mobile geodatabase. I may eventually move my data out of file geodatabases and into mobile geodatabases (which will require an update to all my maps’ sources). And I will begin to offer mobile geodatabases to data-share partners. I think I know what they’ll say.

2020 for 2020

UPDATE 2020-11-14: Done.

2020 for 2020 2020-11-14

2020 for 2020 2020-11-14

2020 for 2020 Strava stats 2020-11-14

2020 for 2020 Strava stats 2020-11-14

***


UPDATE 2020-06-13: Yesterday I inched up ahead of the curve, meaning that I got 45.84% to my goal 44.81% into the year. NerdsЯUs, I know…

2020 for 2020 2020-06-12

2020 for 2020 2020-06-12

***


After having plateaued around 1,500 miles per year for each of the past five years, I aim to ride 2,020 miles in 2020. This is both a physical (I am not 17 anymore) and logistical (full-time job) challenge. A challenge I welcome and embrace, for it helps take my mind off all the craziness that surrounds us. Call it mental escapism.

So while the rest of the world worries about the coronavirus and politics and toilet paper shortage and Bitcoin crashing, I’ll be thinking about ride schedules and training schedules and bike tuning and nutrition and rest. I’ll be making charts and graphs. I’ll be listening to ABBA and Billie Eilish.

You should join me, even if only in spirit.

2020 for 2020 2020-03-14

2020 for 2020 2020-03-14

Reading

[UPDATE 2020-11-02] Not updating this list anymore.

***

Books I have read (or listened to) since returning to book-reading in 2018, in (mostly) reverse chronological order. Date next to each entry denotes date finished (or abandoned).

  • “Jack Reacher’s Rules” by Lee Child ★☆☆☆☆ 2020-10-12
    • I am embarrassed to have read this “book”, which is just a compilation of “guy stuff” clippings from other Reacher books. Stuff like “He was in no imminent danger of winning the Nobel Prize but definitely smarter than the average bear.” and “Female police officers carry handcuffs, which can be handy in bed.” How was Lee Child not embarrassed to publish this? Whose idea was it? I love the Reacher series — good writing, interesting plots. But this “selection” is the worst of the worst.
  • “The Hard Way (Jack Reacher, Book 10)” by Lee Child ★★★★☆ 2020-10-11
  • “Thinking, Fast and Slow” 🕪 by Daniel Kahneman
    • Presently reading.
  • “The Litigators” by John Grisham ★★★★☆ 2020-09-02
    • Solid Grisham.
  • “Robicheaux” by James Lee Burke ★★☆☆☆ 2020-08-24
    • Abandoned.
  • “Bloody Genius” 🕪 by John Sandford ★★★☆☆ 2020-10-01
  • “The Rooster Bar” by John Grisham ★★★★☆ 2020-08-17
    • A lawyer diploma mill scam. Quote: “It’s perfect! It’s beautiful! It’s one great big fat law school scam that’s risk-free. If we default the taxpayers pick up the tab. Rackley gets to privatize the profits and socialize the losses.”
  • “Masked Prey” 🕪 by John Sandford ★★★★☆ 2020-08-17
    • A Lucas Davenport case. Good to listen to during commute.
  • “Echo Burning” by Lee Child ★★★☆☆ 2020-09-11
    • Standard fare Reacher novel. Skimmed through the last 20%.
  • “Running Blind” by Lee Child ★★★★★ 2020-07-03
    • One of the better Reacher novels. Having read most of the Reacher novel series (in random order), and having been disappointed with Lee Child’s recent product, I was pleasantly surprised by this early (2000) work. Well thought-out, well researched, well written.
  • “Murder on the Orient Express” 🕪 by Agatha Christie ★★★★★ 2020-07-08
    • Back to this gem from the queen of mysteries, some 40 years later.
  • “Rogue Lawyer” by John Grisham ★★★★☆ 2020-07-30
    • First Grisham novel in a long time. Enjoyable.
  • “The Mysterious Affair at Styles” 🕪 by Agatha Christie ★★★★☆ 2020-07-09
    • Never read this one before.
  • “Persuader” by Lee Child ★★★☆☆ 2020-06-06
    • Too many words. 544 pages is unnecessarily long for this type of novel. I skimmed through the second half.
  • “The Never Game” 🕪 by Jeffery Deaver ★★☆☆☆ 2020-06-06
    • Abandoned at 25%.
  • “Without Fail” by Lee Child ★★★★☆ 2020-05-28
    • Good novel, overall, with a few oddities: Reacher explains himself too much, uncharacteristically. Reacher wears a watch, despite having an impeccable internal clock elsewhere in the series; phrases like “near as makes no difference”, “laughing up their sleeves”, and “time to time” (notfrom time to time”) appear several times throughout the novel and nowhere else in the Reacher series. Such oddities solidify my suspicions — first arisen by the very poor “Past Tense” — that not all novels in the Reacher series were written by Lee Child himself.
  • “Code of Conduct” 🕪 by Brad Thor ★★☆☆☆ 2020-05-27
    • Abandoned.
  • “Smiley’s People” by John le Carré ★★★☆☆ 2020-05-15
    • Surprisingly, this John le Carré novel did not draw me in as I expected. Too drawn out, too descriptive. Or maybe I have become less patient… Abandoned at 66%.
  • “The President Is Missing” by James Patterson and Bill Clinton ★★☆☆☆ 2020-04-19
    • Lightweight + didactic = annoying combination. Abandoned.
  • “They Shoot Horses, Don’t They?” by Horace McCoy ★★★★★ 2020-04-16
    • Reading this novel some 40+ years after I saw the film was simultaneously inspiring and depressing. I devoured it in two days. 1935 mood same as 2020 mood… “[…] the first existentialist novel to have appeared in America” –Simone de Beauvoir
  • “A Delicate Truth” by John le Carré ★★★★☆ 2020-04-14
    • Private-intelligence service. Military contractors. Politicians. Money. Cover-ups. “So it’s a good old British compromise: a deniable toe in the water but not the whole foot. And me and the boys, we’re the toe, like.” “It’s another merc job, Brigid. Makes you wonder sometimes who starts the wars these days.” A sheer delight, even if somewhat naïve, considering everything that has been happening in the world since the book was published in 2013.
  • “The Sicilian” by Mario Puzo ★★☆☆☆ 2020-04-11
    • A forced, weak attempt to replicate the success of The Godfather. Abandoned.
  • “The Last Don” by Mario Puzo ★★☆☆☆ 2020-04-10
    • A forced, weak attempt to replicate the success of The Godfather. Abandoned.
  • “The Undoing Project: A Friendship That Changed Our Minds” 🕪 by Michael Lewis ★★★☆☆ 2020-05-13
    • Picked up on a friend’s recommendation. An interesting — if needlessly long — account of the collaboration between two brilliant scientists, their contributions to the field of mathematical psychology, and the dynamic of their symbiotic relationship.
  • “The Honourable Schoolboy” by John le Carré ★★★☆☆ 2020-04-14
    • Surprisingly, this John le Carré novel did not draw me in as I expected. Too drawn out, too wordy. Or maybe I have become less patient… Abandoned.
  • “Interior Chinatown” by Charles Yu ★★★★★ 2020-05-20
    • Delightful. You don’t have to be a Chinese immigrant in America to understand, appreciate, enjoy, love this warm, thoughtful, witty, beautifully written narrative.
  • “Annihilation” 🕪 by Jeff VanderMeer ★★★☆☆ 2020-03-06
    • Clever sci-fi novel, whose elaborate descriptions of EVERYTHING — though masterful — felt a bit too elaborate for me. The whole thing could have been a novella, a short story even. It could also be that I am not all that much into sci-fi in general.
  • “Blue Moon” by Lee Child ★★★★☆ 2020-03-01
    • Back to Lee Child and Jack Reacher, after getting over my serious disappointment with one of Child’s recent Reacher novels. This one is so far so good. It appears that Child has bounced back from his recent slump (or whatever else caused him to put out sub-par work). // This is good writing. It is amazing, almost inexplicable, how Child’s writing quality can vary so greatly from Reacher novel to Reacher novel. // Hmmm… Albanian gangster using phrases like “opportunistically and treacherously”? I will withhold further judgement until I finish the book. // A good Reacher novel. Interesting plot twists — some predictable, some not. Above all, it’s good to have back the Child I knew (who seemed to have gone missing in some of his recent works, most notably “Past Tense”).
  • “Trigger Warning: Short Fictions and Disturbances” by Neil Gaiman ★★★★☆ 2020-03-14
    • Came for the short fiction, found disturbances. Abandoning for a lighter fare, for I have enough disturbances in my life as it is.
  • “Suzy Spitfire Kills Everybody” by Joe Canzano ★★☆☆☆ 2020-02-21
    • I bought this book because I wanted to support the author — a friend whom I know as a musician. I didn’t know what to expect from his writing. Fifteen pages into the novel I am hooked. Fast-paced sci-fi action amid witty dialog, taking place in the (not-so-distant?) future when the United Mexican Union is one of the nine nations on Earth. Quote from page 16: “He obviously wasn’t expecting anyone hostile, like a dentist.” // 90 pages into the 295-page novel the wisecracks and the one-liners begin to wear thin. I put the book down. Maybe I’ll get back to it and finish it at another time.
  • “Charting and Technical Analysis” by Fred McAllen ★★★★☆ 2020-02-16
    • First book bought with Bitcoin. Actually, first-ever Bitcoin purchase. // Learning about stock trading and charts and trend lines and patterns and TA (technical analysis), and forecasting, and lemme tell ya: Voodoo is much more scientific than any of that. But I will finish the book, and I will use my new-found knowledge in my cryptocurrency adventures. // I abandoned the book on page 114, at 43%. The book is written at a 9th grade level, which is probably what its target demo is. I bought this book because I wanted to make sense of the mumbo-jumbo on crypto Twitter, in which I became suddenly, (and briefly, as it were) interested. I also wanted to buy something with Bitcoin, and this book was a perfect candidate. It did its job for me, and gave me what I was looking for. I may return to it if and when I decide to become a day trader or swing trader. For now, I am just a HODLer.
  • “Finders Keepers” 🕪 by Stephen King ★★★★★ 2020-02-22
    • First King book in a very long time. // A well-woven story about an obsessive literary fan, with all the crime and drama and suspense and characters King is known for. It kept me looking forward to my commutes.
  • “The Hitchhiker’s Guide to the Galaxy” by Douglas Adams ★★★★★ 2020-02-12
    • What took me so long to get to this book??? Quote (this is science fiction written in 1978, mind you): “Only six people in the Galaxy knew that the job of the Galactic President was not to wield power but to attract attention away from it.” Also: “There are of course many problems connected with life, of which some of the most popular are Why are people born? Why do they die? Why do they want to spend so much of the intervening time wearing digital watches?” Also: “It is an important and popular fact that things are not always what they seem. For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much—the wheel, New York, wars and so on—while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man—for precisely the same reasons.”
  • “Stealth” by Stuart Woods ★☆☆☆☆ 2020-01-21
    • A new low on my list. A wealthy NYPD detective owns an estate in the UK across the river from the estate of the female director of MI-6, with whom he has sex. He next has sex with an attractive female doctor, who is also an MI-6 agent. Russian paratroopers arrive during the night on black parachutes. Abandoned the book on page 56. Should have abandoned it sooner.
  • “Camino Island” by John Grisham ★★★☆☆ 2020-01-20
    • Rather bland story set in the world of books, writers, booksellers, and stolen F. Scott Fitzgerald manuscripts. Main takeaway for me: the advice of a seasoned bookseller to a young novelist about what not to do in her novels. Not Polonius-level stuff, but interesting and potentially useful nonetheless.
  • “The New Girl” 🕪 by Daniel Silva ★★★★☆ 2020-01-24
    • The obligatory filler notwithstanding, this is one of the better and more interesting Silva books in the Gabriel Allon series. Loosely based on the killing and dismemberment of Saudi Arabian journalist Jamal Khashoggi in the Saudi consulate in Istanbul, the author fictionalizes the events leading to and following the murder, with an interesting twist of events.
  • “Make Something Up: Stories You Can’t Unread” by Chuck Palahniuk ★★★★☆ 2020-01-10
    • Having belatedly discovered transgressive literature, I was both fascinated and disturbed by this collection of short stories. I didn’t finish it. Definitely not everybody’s cup of tea. ‘Zombies’ is a gem.
  • “The English Spy” by Daniel Silva ★★☆☆☆ 2020-01-12 // 🕪 
    • Presently reading. I am having trouble finishing this one, either because I have little interest in the Northern Ireland / UK / IRA conflict, or because there’s just too much filler. // Picked up this novel again as an audio book, to listen to during my commutes. // Paused.
  • “Mr. Penumbra’s 24-Hour Bookstore” by Robin Sloan ★★★★★ 2019-12-24
  • “An Absolutely Remarkable Thing” by Hank Green ★★★★★ 2019-12-10
    • Futuristic sci-fi with present-day characters. Sheer delight.
  • “The Heist” 🕪 by Daniel Silva ★★★☆☆ 2020-01-04
    • Like most Daniel Silva novels, this feels unnecessarily long.
  • “Permanent Record” by Edward Snowden ★★★★☆ 2019-10-31
    • This book did not tell me anything I didn’t already know or suspect, yet it caused me to reassess and alter my online behavior. The first ⅔ of this book is brilliant and dense, then things trail off in the last ⅓. Everyone should read this book, regardless of their personal opinion of the author.
  • “Die Trying” by Lee Child ★★★☆☆
    • I am having trouble finishing this one. Unnecessarily wordy, it feels as if the author is getting paid by the page. Maybe I’m just getting tired of Jack Reacher.
  • “The Black Widow” 🕪 by Daniel Silva ★★★★☆ 2019-12-10
    • Israeli uber-operative Gabriel Allon discovers, follows, and neutralizes an Islamic terrorist cell. Good for four weeks of commuting.
  • “Agent Running in the Field” by John le Carré ★★★★★ 2019-11-25
    • John le Carré’s latest novel, masterful as always. However, as someone who has read most (all?) his work, I can’t help but feel that he is running out of material.
  • “The 14th Colony” 🕪 by Steve Berry ★★☆☆☆ 2019-11-08
    • Bloated. Why is every recently-written fiction / adventure book over 400 pages? Do authors really get paid by the page? So much filler…
  • Gathering Prey
  • Neon Prey
  • Golden Prey
  • False Friend
  • False Witness
  • “The Mission Song” 🕪 by John le Carré ★★★★★ 2019-09-26
  • “Our Kind of Traitor” 🕪 by John le Carré ★★★★★ 2019-09-03
  • “A Legacy of Spies” by John le Carré ★★★★★ 2019-08-15
  • “The Pigeon Tunnel” by John le Carré ★★★★★ 2019-08-04
    • A beautiful collection of true stories as experienced and told by a field operative thinker
  • Death of an Honest Man
  • Death of a Kingfisher
  • Death of a Liar
  • Death of a Ghost
  • 61 Hours
  • Personal
  • Not a Drill
  • The Christmas Scorpion
  • Death of a Maid
  • No Middle Name
  • Never Go Back
  • High Heat
  • “The Fourth K” by Mario Puzo ★★★☆☆ 2019-05-18
    • Mario Puzo does a political thriller. It’s OK, but I keep waiting for the mafia to show up and it doesn’t.
  • Enlightenment Now
  • 21 Lessons for the 21st Century
  • Nothing to Lose
  • The Affair
  • The Murderer’s Daughter
  • Motive
  • Killer
  • Guilt
  • Victims
  • Deception
  • A Wanted Man
  • “Past Tense” by Lee Child ★★☆☆☆
    • The poorest of all Reacher novels. I strongly suspect it was not written by Lee Child himself but by a novice ghostwriter. Many similar sentiments appear in Amazon reviews.
  • Bad Luck and Trouble
  • Jack Reacher: One Shot
  • Worth Dying For
  • Make Me
  • Small Wars
  • The Enemy
  • Night School
  • Deep Down: A Jack Reacher Story
  • The Midnight Line
  • Gone Tomorrow
  • Killing Floor
  • “And Then There Were None” by Agatha Christie ★★★★★ 2018-12-16
    • I first read this half a century ago in Bulgarian. It was called “Десет малки негърчета” — “Ten little negroes”. I have since read it many times in English as “Ten little Indian boys”, “Ten little sailors”, and now as “And Then There Were None” — the publishers progressively yielding to the PC demands of the day. Regardless, this is a superb mystery, and I will probably read it again in the future.
  • “The Murder of Roger Ackroyd” by Agatha Christie ★★★★★ 2018-12-09
    • I first read this half a century ago in Bulgarian. It was called “Алиби” — “Alibi”. I have since read it many times in English. A superb mystery from the queen of mysteries. Allegedly Christie wrote this novel as a response to a friend’s claim that murder mysteries are transparent and it is always apparent who the murderer is.
  • Judgment Cometh
  • A Crime of Passion
  • Conflict of Interest
  • Blood Money
  • Reasonable Fear
  • Injustice for All

Future reading:

  • “The Shallows” by Nicholas Carr
  • “Indistractable” by Nir Eyal
  • “The Looking Glass War” by John le Carré
  • “The Ministry of Fear” by Graham Greene
  • “String Theory: David Foster Wallace on Tennis” by David Foster Wallace
  • “Homo Ludens: A Study of the Play-Element in Culture” by Johan Huizinga
  • “Bitcoin Billionaires” by Ben Mezrich
  • “Attack of the 50 Foot Blockchain” by David Gerard
  • “Tao Te Ching”

Confronting my technological prejudices

I have always been superficially prejudiced against all things Apple. When a seasoned technologist friend announced switching from Android to iPhone, I was curious. I asked him why. It was illuminating for me to hear his reasoning, which I present to you below.

***

When I saw that my friend Bill Dollins — a long-time Android and Linux guy — is now an iPhone user, I was somewhat surprised. I wondered why he switched. I swiftly jumped to my own conclusions about his reasoning. Then, realizing that my conclusions were probably colored by my own biases (they were), I decided to just ask him. Bill agreed to entertain a few questions.

AT: Bill, why did you switch from Android to iPhone? I thought tech people use Android as a badge of honor. I thought Apple products were for artists, celebrities, and the VP of Marketing.

Bill: Working in an Apple shop allowed me to learn more about their products than I would have on my own. My primary reason has become privacy and security. The contrast on these issues between Google and Apple is stark. Apple takes these issues much more seriously for individuals than does Google and it is evident in technical and business choices they have made. 

For example, Apple’s smart home technology, HomeKit, doesn’t specifically require internet access, though some individual devices may. They do things like video processing via a HomePod, which can be disconnected from the internet. These are functions that Google or Amazon offload to the cloud, so connectivity is required. This design choice means that HomeKit doesn’t yet have a video doorbell option, so it can “slow down” feature development when compared to Google or Amazon. I’m okay with that.

The iPhone takes a similar approach. Like Android, it has a location history feature. Unlike Android, the history and all related processing remains on the device. There’s no centralized Apple cloud where it’s being stored along with everyone else’s to train some machine-learning algorithm.

I should note that I haven’t done anything yet with HomeKit myself, but I am considering it. My information comes from my Apple consultant, James Fee.

Having said all of this, I know that no technology is perfect. Zealots of Googlism or Amazonism will certainly nitpick anything that I have said here. That’s what block buttons are for.

AT: Where do you stand on the Windows / Linux / Mac divide for desktops and laptops?

Bill: My primary desktop machine, which I use daily, is still a System76 Ubuntu machine. My MacBook is company-issued and mainly used for specific work tasks and for when I travel. Windows has been the big loser for me.

AT: I used a MacBook once, years ago. I remember apps performing an elaborate dance on open and close. It made me dizzy, and I thought it was childish. I haven’t touched one since. Do Macs still do that?

Bill: I haven’t noticed that since I have been using a Mac. One thing I did notice is that some Mac idioms, such as two-fingered touchpad scroll, exist on more modern Windows machines alongside the older Windows idioms. I have an older Windows laptop that doesn’t support it, but I haven’t looked closely to see if that’s because of the generation of the hardware or of Windows. I’m not really concerned about it enough to investigate.

AT: Don’t tell anyone, but I have been thinking about an iPhone for a while, mainly as a means to distance myself from Google. I haven’t done it yet because I am concerned that my reputation will take a hit. What say you to that?

Bill: I will defer to the expert on this topic: Joan Jett. 

https://www.youtube.com/watch?v=nO6YL09T8Fw

Let’s all go back to blogging

I’ll go first.

I went for a bike ride yesterday. Nothing new, except this time I allowed myself to break away from the pursuit of performance metrics (speed, distance, etc.), and focused instead on the journey (autumn, sun, cool air, trees). What a difference! I still hit some decent numbers, but that was secondary.

Here’s a pic from a break I took along Canal Road and the Delaware and Raritan Canal in Franklin Township, New Jersey:

Autumn_biking_on Canal_Road

Some formats are more legacy than others

Having promised to myself (and to whomever listens on Twitter) that I would not think about GIS over this long Labor Day weekend, I did nothing but. Why? A nagging issue that stems from my earlier effort to abandon ArcMap in favor of ArcGIS Pro. While most of my workflows port nicely to Pro, one of my workflows does not work at all. That workflow includes periodic MS Access downloads from a proprietary third party application and joining Access tables to “the GIS”.

It took the cumulative advice of a Twitter village over the Labor Day weekend to convince me that the weakest link in my workflow is MS Access, and that I should focus on liberating my data from Access tables instead of blaming Esri for dropping Access support in ArcGIS Pro.

In the next few days I will experiment with a number of Access-liberating workflow remedies, and will document my experience by adding to this post. Suggestions I have received so far:

  • Use FME
  • Install driver for Excel 2019 and copy/paste data from Access tables to Excel, then join the Excel table
  • Export the Access table to CSV or TXT, then create table(s) from the text file(s)
    • Either via the Access GUI or write a Powershell script to export the Access data to CSV
  • Use Airtable
  • Write Python script to export Access data

Many thanks to (in order of appearance): Simon Jackson, Craig Williams, Bill Dollins, Heather S, Jason Birch, James Fee.

***

[UPDATE September 4, 2019]: What I did today:

  1. Downloaded a 30-day FME trial. Looks great, but procuring $2,250 might be a challenge. Possible long-term solution.
  2. Downloaded the Excel driver, on install got this error. Graig Williams promptly offered a workaround, which worked. Copied Access data into Excel sheet, added Excel sheet to ArcGIS Pro, and joined the resulting table to my shapefile. Everything worked as desired.
  3. Exported the Access table via the Access GUI to a text file (named .CSV), added CSV file to ArcGIS Pro, and joined the resulting table to my shapefile. Everything worked fine, except the table truncated my leading zeros in fields such as ZipCode. I’m sure there is a simple fix.
  4. Looked at Airtable. Looks promising, but both the cost and the “data in the cloud” aspects make it a non-starter for this task.
  5. Did not write Python script. 

Going forward I will follow the steps in item 2 above, using the tools I already have. I will continue to explore ways for liberating Access data, up to and including writing my own scripts. I no longer blame Esri for dropping Access support from ArcGIS Pro.

When you reach a trident in the road, take it

Many thanks to Bill Dollins, Craig Williams, and Randal Hale, whose generous counsel and thoughtful insights continue to guide me along my GIS journey.

I am facing the desktop GIS trident: ArcMap, QGIS, ArcGIS Pro. Three tines. Which one should I take?

In my earlier cycling metaphor I compared selecting bicycle pedals to selecting GIS software tools. The pedals debate is over (Speedplay won); the GIS debate still rages on. I have analyzed and overanalyzed. I have consulted with the wisdom of crowds (Twitter) and the wisdom of individuals (Twitter DMs). I have considered various platforms, data formats, and multiple combinations and permutations thereof. It is time to wrap it up.

“I hear and I forget. I see and I remember. I do and I understand.” –Confucius

I begin with a 30-day challenge: use nothing but ArcGIS Pro for all my day-to-day GIS activities. I will document my experiences — good and bad — and I will update this post continuously with a running tally of said experiences. I will sum it all up on March 15.

Here we go.

  • DAY ONE [1], February 14, 2019: The challenge begins. QGIS and ArcMap and ArcCatalog icons stare at me from the Windows 10 taskbar. I do not click on them, which feels weird. I click on the ArcGIS Pro icon next to them. I am greeted with a warning: “Pro Advanced will expire in 14 days.” Stress levels go up. We always pay our bills, but sometimes we are late. What will happen on March 1 if our P.O. doesn’t make it to Redlands in time? ArcMap and QGIS never give me that kind of stress. At least all my data is on our local network. I click OK. ArcGIS Pro 2.3 launches. I single-click on a recent project. Project loads. I see my map. Some of my data is in a file geodatabase (FGDB), some in shapefiles (SHP). At some point I will need to replicate a Desktop process to periodically export a FGDB feature class to SHP and manipulate said SHP (delete and add fields) before I upload the SHP into a proprietary system. I’ll also need to replicate a separate process where I join an SHP to an Access database. I’ll attempt all of that next week. Today I’ll just tackle some light tasks. I create a custom basemap from custom aerial photography. It works. I merge two polygons. The process is smooth, similar to ArcMap’s. I hit the Save button. Does it save the project, or the merged polygons?
  • DAY TWO [2], February 15, 2019: “Pro Advanced will expire in 13 days.” Okay then. I open the same project. My custom base map is gone, even though before shutting down yesterday I clicked every Save button I could find. What did I miss? OK, it’s right there in the documentation — the custom base map has to be saved in the project you want to use it in. You can’t share custom base maps across projects. I recreate the same custom basemap in the current project. On to Catalog. Add Folder Connection works just like on Desktop. It does not auto-refresh, so if you add a new file to a connected folder you won’t see the file until you hit Refresh. Deleting fields in SHP took considerably longer than the same process in Desktop, but not a big deal for me. Adding fields is similar to Desktop, while the ability to add more than one field at a time is a welcome improvement. Now let’s join this puppy to an MS Access table. Wait, what? No MS Access support in Pro? I found a discussion on an Esri support site, basically saying don’t use Access. I beg to differ. One of my workflows is constrained by hard external factors, and one of these factors is “use Access”. Another is “use SHP”. So I will continue to use Access and SHP. But I should be able to find a workaround. Export Access table do DBF doesn’t work because long field names get truncated, effectively resulting in trying to have the same field name more than once in the same table, which DBF doesn’t like. Export to Excel and then join SHP to XLSX works. Whew! The world’s most popular database saves the day.
  • DAY THREE [6], February 19, 2019: “Pro Advanced will expire in 9 days.” Got it. I click OK. ArcGIS Pro crashes. Should I send a report to Esri? I don’t always do, but I decide to send a report this time. Report sent. Launching Pro again. “Pro Advanced will expire in 9 days.” OK. Pro runs. First I address a blog comment question from Nancy von Meyer: “Can you make a starting project with your custom base map and save that with just the base map then open it and save as something different and have the custom base map in the saved as project?” I try this. Yes, you can, and this is a convenient way to save and reuse your custom basemap. Thank you for the suggestion, Nancy! Next I decide to tackle a monthly GDB maintenance task — “Compact”. (There is also “compress”; find the difference between compact and compress on your own.) It is unclear whether it’s safe to run Compact while having the same GDB in a map in the same project. I decide to run the process on a scratch GDB and see if something blows up. Compact runs, nothing blows up. I reindex a feature class in the scratch GDB. Nothing blows up. Save project. Close Pro. Open Pro. All my stuff is still there — custom base map, recently-compacted and reindexed scratch GDB, production GDB. On to compact and reindex the production GDB. Smooth sailing. (In recent weeks I have been getting error messages while editing and saving this same GDB in ArcMap. Errors were blamed (incorrectly, IMHO) on bad network connection. Very curious to see whether the same errors on the same GDB will crop up in Pro. So far they have not. Good!)
  • DAY FOUR [7], February 20, 2019: Day of routine tasks, already written up in prior days. No surprises — good or bad. Pro just works.
  • DAY FIVE [8], February 21, 2019: Annotation. Everyone’s got a pressure point, and mine is annotation. I know — not sexy, unhip, lame. Nothing like 3D or a billion points in a database in the cloud. But annotation is what brought me to the GIS trident. Specifically, the need to update and maintain going forward a fairly complex map for print, some of whose sources are personal GDB annotation feature classes (don’t ask). The map — some of whose layers I have inherited and have built upon — gives me stress every time I touch it. Will something blow up? I hope that moving everything to a fresh platform will make this stress go away, and the transition will have been worth it — both technically and psychologically. But do I move (or recreate) these feature classes to ArcMap, or QGIS, or Pro? Hence the GIS trident. Hence this crucial point in the 30-day ArcGIS Pro challenge, the acid test: Will Pro and its annotation capabilities rise to the occasion? Let’s find out. Add road centerline layer to map; zoom to desired scale (determined by trial and error with previous map); label roads; play with symbols and position. Everything works as expected, although redraws take a tad longer than I am used to. I can live with that. Convert To Annotation. “Create feature-linked annotation” shows a warning — option is only available with ArcGIS Desktop Standard and ArcGIS Desktop Advanced licenses. Clearly Pro is not separate from Desktop. I do have a Desktop Advanced license, so I proceed to run the process. Error. Feature-linked annotation can only be created in the same geodatabase as the base features. Repointing target destination. Run. Same error, even though I redirected the output anno to the same GDB as the base. Save everything. Quit Pro. Start Pro. Run Convert to Annotation again. Process fails with the same error. Oh, well… Nobody said it’d be easy. Easy is for amateurs. Could it be that even though I have Desktop Advanced, Pro doesn’t KNOW that I have Desktop Advanced because the two are licensed differently and don’t talk to each other? I’ll tackle that tomorrow.
  • DAY SIX [9], February 22, 2019: Annotation, day two. We have just paid our Esri bill and the licensing expiration warning message is gone. Could this have caused the anno error? I uncheck the “Create feature-linked annotation” and hit Run. The process runs. I check it — the process fails. What else could be causing the problem? The level of my license, or something else? Craig Williams offered several suggestions (thank you, Craig!), and one of them resolved the issue. It turns out that “Create feature-linked annotation” attempts to create annotation from all labels in the map. As it were, I had in the same map a shapefile-based layer, labelled, which was the culprit. As soon as I turned off the shapefile layer and its labels, Pro created feature-linked annotation from my intended road centerline layer’s labels. Pro is very close to passing the acid test. Next step in the anno test: manually edit annotation. I’ll tackle this next week.
  • DAY SEVEN [12], February 25, 2019: An unscheduled task takes precedence over the planned anno editing: Georeference a scanned image and digitize over it. Georeferencing and subsequent polygon digitization and attribute table editing are smooth and straightforward. BTW, the attribute editing on this same FGDB is where ArcMap would give me the occasional weird error. No errors in Pro so far. However, it is not always clear what I am selecting (I inadvertently deleted a layer from my map instead of a polygon from said layer), but I’m sure I’ll get the hang of it with practice. To resolve the inadvertent layer deletion (no “undo”), I opted for quitting Pro without saving the most recent action. Pro took a VERY long time to close the project and quit. After reopening Pro I open the same project. All my stuff is still there. Whew! I was saving the following comment for the summary at the end of the challenge, but I’ll jump the gun and say it now. I understand why Esri needs to sunset ArcMap. I appreciate all the new development that went and continues to go into Pro. But sometimes I wish Esri would put all the new Pro tech behind the familiar workflows of ArcMap, split the Pro product into Map and Cloud, and give the map-makers and cloudsters all the tools each group needs and none of what they don’t. Because, to quote myself: “The intersect of #ArcGISPro users who make maps for print and those who need billions of points in a database in the cloud is NULL.”
  • DAY EIGHT [13], February 26, 2019: Got a request to create a shapefile (“Ask for it by name!”) from a text file with coordinates. Pro delivers — creates a FGDB feature class first, which I then export to shapefile to satisfy the SHP delivery format request. Another unscheduled task takes precedence over the planned anno editing: Update one in a series of 30 PDF (of MXD lineage) maps. The map update itself should be trivial, but this will present a less trivial logistical / data management dilemma: What to do with the rest of the maps in the set? Don’t touch them and end up with a hybrid (MXD / APRX) map set, or convert them all to APRX and effectively predetermine the 30-day challenge outcome? I decide to cheat and update the map in ArcMap — not because Pro can’t handle the task, but to not contaminate the outcome of the 30-day challenge. I violate the rules of the challenge in order to keep it pure.
  • DAY NINE [14], February 27, 2019: Day of routine tasks, already chronicled in previous days. Pro just works, I am getting the hang of it. The crashes are gone, as is the license expiration warning (causality between crashes and impending license expiration unclear). No FGDB save errors, either. Overall a very smooth experience, if you don’t count the hunting and pecking for functions you know are there but who are seemingly playing hide-and-seek with you.
  • DAY TEN [15], February 28, 2019: Another day of routine tasks, so I’ll pontificate about credits consumption. Last year I inadvertently and accidentally (and quite quickly and easily) consumed a large number of service credits, just by clicking the wrong button two or maybe three times (Esri acknowledged my mistake and graciously refunded the account). Ever since I have been keeping a daily eye on the account balance, to make sure I have not possibly accidentally clicked on that (or a similarly-credits-draining) button again. The credits price list is long and confusing. It is mostly clear and makes sense while I read it, but a few days or weeks later doubts inevitably creep up (“Does this geoprocessing button consume credits?”) and I have to pause and check the price list again. It would be very nice if Pro unambiguously, clearly, and loudly informed users about credits consumption — up to and including putting the free geoprocessing tools behind a completely different section of the ribbon than credit-consuming functions.
  • DAY ELEVEN [16], March 1, 2019: Midway upon the journey of my 30-day ArcGIS Pro challenge I remind myself that the purpose of the challenge is to test whether I can replace Map with Pro. Not a comprehensive review of Pro’s capabilities. To put it bluntly: Can I stop using Map and still do my job. That is all. To that end I will not be testing integration with ArcGIS Online (AGOL), for this is not a Map function I’ll need to replace. Apropos, I have been using Pro to author online maps since 2016. Seven of these maps are shared with the public in an online portal. The process is not without its quirks, but workable. Credit consumption is minimal (over the course of one year I have consumed 24.01% of the complimentary credits that come with the annual  Esri maintenance). But enough about AGOL. Back to the daily grind where an internal customer calls for help with creating a list (in Excel) of property owners based on a fairly complex spatial selection. A breeze for Pro up until exporting the selected attribute data record set, where copy and paste into Excel just works, except Excel truncates the leading zeros in the zip code column (this is NJ, y’all), so I end up using the Excel CONCATENATE function to tack on a zero in front of each now-four-digit ex-zip code. On second thought, I could have pre-configured the receiving Excel spreadsheet column as text. I’ll try to remember next time. On that note, have a nice weekend, and see you Monday!
  • DAY TWELVE [19], March 4, 2019: Once per month I take a snapshot of my GIS data and upload it to a proprietary application accessed by many users. While I am free to choose the format in which I store and maintain the master GIS dataset (FGDB, PostGIS, coverage (I would if I could), whatever), the monthly snapshot requires SHP and Access, non-negotiable. Today and tomorrow (it’s a two-day process) will tell whether Pro can replace Map in this vital step. First (minor) hiccup: Unlike Map, Pro won’t write a shapefile with dashes in the filename. Dashes are useful if I want to include an ISO-formatted date in the filename, like so: “Description_2019-03-04.shp”. Should I eliminate the dashes or replace them with underscores? I opt for underscores over the ISO-compliant “20190304”. The process runs in the Geoprocessing panel, causing a brief blood pressure spike. “Does this consume service credits?” The new shapefile is automatically added to my map, want it or not. Next I need to delete some fields from the new shapefile’s attribute table. I should probably remove the layer from the map first (old skool lessons learned). I do that, and the Drop Field process runs successfully. Would it have run with the layer still in the map? I’ll try that some other time. One difference from the Delete Field process in ArcMap/Catalog is that in the Drop Field panel Pro rearranges the field names alphabetically — uncalled for and confusing, IMHO. Next I add several new fields to the SHP — the process is smooth and straightforward. So far so good.
  • DAY THIRTEEN [20], March 5, 2019: Import SHP into FGDB feature class. This is a big step, psychologically. Because this SHP is an export from a QGIS layer living in PostGIS. And that layer ended up in PostGIS because ArcMap had been giving me random, inexplicable errors on saving in its previous home — an FGDB. Which is one of several reasons that brought me to the trident — the ArcMap/FGDB combo had become unreliable, so I sought refuge in PostGIS (edited alternately by QGIS and GISquirrel on top or ArcMap). However, at this point in the 30-day challenge, I feel confident enough to bring my precious data back into an FGDB, this time managed in Pro. The ultimate goal being that, as Einstein allegedly said, “Everything should be made as simple as possible, but not simpler.” On a different note, tomorrow I will do my best to join a new SHP to a new Access table, as part two of that two-day process. Stay tuned.
  • DAY FOURTEEN [21], March 6, 2019: Part two of the abovementioned two-day process. Export SHP and Access from proprietary app, add SHP to Pro map, export Access table to Excel, add Excel table to Pro map, join SHP and Excel, import newly-joined SHP into FGDB feature class, done! Check for no-matches: Select by attribute where ‘fieldname’ is null. Bingo! Very pleased with the whole process. This was a major step in the 30-day challenge, and a deciding factor in whether Pro will get the thumbs up from this critic. Next, another export FGDB -> SHP -> ZIP to share with data partners (who ask for SHP by name, FTR). A noticeable difference — the resulting ZIP file size is 39% larger than that of a similarly-processed and -compiled file using ArcMap. Moving right along. Once again, just like I did on DAY SEVEN, I mistakenly delete a layer from the map instead of deleting a single selected feature in that layer. Am I obtuse? Also (indirectly related) I can select a feature and delete it while in the Map tab, but to save the edit I must first click on the Edit tab in order to expose the Save button which is otherwise hidden. Very convoluted.
  • DAY FIFTEEN [22], March 7, 2019: Editing in Pro all day. Lots of improved tools. I love it. No complaints today. I have been twitter-following the Esri Developer Summit, and I see all these new exciting things, and I see all these users tweeting excitedly about all these new exciting things. And I want to tell them: GIS is a marathon, not a sprint. By the time you get around to implementing these exciting new things, many of them will be obsolete or discontinued. Play the long game. Don’t turn up your nose on innovation, but don’t turn your back on the shapefile, either.
  • DAY SIXTEEN [23], March 8, 2019: Import MXD into Pro = Insert Map (MXD) into current project. Pro imports the Layout View, true to the original. Export to PDF (Share Layout) works as expected. 80% of GIS functionality covered right there. Or possibly 90%. I wonder if I can import the ArcMap Data View. Yes! Apparently Pro imports the state of the MXD in which it was last saved. Save the Layout View — import the layout. Save the Data View — import the “Map”. Am I ready for the Map-to-Pro transition, one week early? Possibly, but no reason to rush. Hang around for one more week, and read a summary of the 30-day challenge next Friday, March 15, 2019. Have a nice weekend!
  • DAY SEVENTEEN [26], March 11, 2019: Today I intended to import and edit FGDB annotation and Personal GDB annotation from ArcMap. Then I realized that I have become quite comfortable with labeling and creating feature-linked annotation in Pro. The much improved label placement algorithms and extensive set of user controls for labeling leave very little need to edit the new annotation. I decide that it would be more efficient to recreate all needed annotation in Pro than to try to salvage the existing ArcMap FGDB and personal GDB annotation.
  • DAY EIGHTEEN [27], March 12, 2019: Will an ArcMap custom tool work in Pro? In ArcMap I have been using a Python (ArcPy) script (Desktop ArcToolbox tool) that selects streets that intersect with grid, creates a street index, and exports to PDF. A nice, dressed-up spatial join. I found this tool (developed by Wes Miller) years ago on an Esri support page that no longer exists. Will this code work in Pro (with or without tweaks)? I drag and drop the TBX into my Tools folder. I launch the tool. I fill in all required parameters, the tool runs for a while, then throws an error, understandably. Line 52: “mxd = arcpy.mapping.MapDocument(mxdDoc)”. Should I tweak the code to make it work in Pro? On brief reflection, I decide to just run a straight spatial join. I do, it gives me the output I need. Stay as close to the core as possible. One less moving part. I congratulate myself for simplifying my life.
  • DAY NINETEEN [28], March 13, 2019: As the 30-day challenge winds down, I am running out of new things to try. While I am sure new types of tasks will present themselves in due course, for now I have covered pretty much all aspects of my regular workflow. I realize that there are vast areas of Pro functionality that I haven’t touched –3D, raster analysis, network analysis, etc. I will tackle those if and when the need arises.
  • DAY TWENTY [29], March 14, 2019: On this penultimate day of the 30-day ArcGIS Pro challenge I ruminate quietly on the ideology and practicality of software choice. I will leave off the long of it for a future tractate. The short of it is: Should I also do a 30-day QGIS challenge? After much thought I decide that this would only be necessary if I determine that Pro can not fully replace Map in my daily duties. The trident described in the beginning of this post is reduced to a bident, with QGIS lying in wait for Pro to fail. Will Pro fail? Tune in tomorrow for the 30-day ArcGIS Pro challenge summary.
  • DAY TWENTY ONE [30], March 15, 2019: The 30-day ArcGIS Pro challenge ends.

What did I learn?

TL;DR

The 30-day ArcGIS Pro challenge was designed to answer one question: Can I replace ArcMap (Advanced) with ArcGIS Pro (Advanced) and still do my job? The answer is yes.

FACTS

  • ArcGIS Pro works.
  • As of today (March 15, 2019) Pro has all the functionality I need to do my daily job.
  • Pro is being developed actively.
  • Pro refreshes the OSM basemap practically instantly
  • The imagery basemap (out beyond ~3,000 scale) is presently the most up-to-date basemap out of the four basemaps I use across all applications
  • Pro crashes occasionally on launch (rarely).
  • Pro doesn’t support MS Access.

OPINIONS

The Pro crashes don’t bother me (as long as Pro doesn’t take down my data with it, which it does not). I trust that the crashes will go away as the product matures and stabilizes.

ArcGIS Pro is complex. Unnecessarily complex, for my needs. At times I feel like I am driving an 18-wheeler to the corner deli to get a sandwich. A four-wheeler will do, or even a two-wheeler. I wish Esri would have split Pro into the equivalents of 18-wheeler, pickup truck, passenger car. Or ProMap for map production, ProCloud, Pro3D…

Pro is confusing. Probably less so for someone just getting her feet wet in GIS, but definitely for someone used to the ways of ArcMap and ArcInfo Workstation and PC ARC/INFO and ArcView 3 and ArcView 2 and ArcView 1. Many times during this trial I wished Esri had put the new technology developed for Pro behind the familiar workflow of ArcMap. Not the interface — I don’t mind a new interface — but the workflows ingrained in any seasoned geogeek’s subconscious.

WISH LIST

MUST HAVE:

  • MS Access support
  • Save Edits button visible any time an edit function is performed

NICE TO HAVE:

  • Different symbols/icons in the table of contents depending on source type, like in Catalog (SHP vs FGDB, etc.)
  • Recalculate map extent as new layers are added or removed.
  • Make the Export function more user-friendly. Currently I can export to SHP or DBF, but I must type the file extension to make it happen. Not really intuitive. By default all exports go into FGDB. It would be nice to have a drop-down to specify SHP, DBF, etc.

TIPS

  • Compact your FGDB(s) monthly
  • Reindex your FGDB feature classes monthly (Recalculate Spatial Index)

SUMMARY

ArcGIS Pro has all the functionality I need to do my daily job. I am making Pro my default GIS tool, effective immediately. Over the course of the last month (21 business days) I have grown familiar with and even fond of Pro. However, the full transition from Map to Pro will not be instantaneous. I will not be uninstalling ArcMap from my PC anytime soon.

Pro gave me the confidence (and nudge) to clean up my data and streamline my workflows. It prompted me to ask (and answer) questions like: How much of the past do I need to preserve? What is important and what is not? Obviously I want to preserve my core data, like road centerlines. But what about derivative data, e.g., annotation derived from those centerlines? Knowing that some of those annotations have been tortured into submission to fit into their allotted spaces, is it reasonable to expect that they will move over smoothly to the new platform? Does it not make more sense to recreate them from scratch? Ditto for some cosmetic customizations, which are nice to have but not crucial for my workflow — drop them and stay close to the core?

Pro taught me humility. I jumped into it, cocky, flaunting my 28 years of GIS experience. I stumbled many times. I learned from my mistakes, sometimes while cursing.

It is a virtual certainty that new types of GIS tasks will come my way in the future — tasks I did not have the opportunity to test during the 30-day challenge. I am reasonably confident that I will be able to handle them in Pro, and I look forward to the challenge.

Finally, what about QGIS? Is it not a great tool? I have been using QGIS for many years. I have come to appreciate its growing functionality, the effort of its developers, and the concept of freedom that it espouses. I am well aware that QGIS outperforms ArcMap / Pro in some areas. I am also aware that there are multiple factors into measuring performance, workflow compatibility being a major one. At my current station in life and career, embracing QGIS as my primary tool is just not practical.

Thank you for reading thus far. I hope this write-up has been useful. Have questions? Leave a comment, or hit me up on Twitter, or email me at atanas [at] entchev.com.

On bikes and GIS

On bikes and GIS

I have been thinking a lot about bikes lately. Specifically, about transitioning from a cross bike to a road bike. More specifically, about transitioning from cross pedals to road pedals. Even more specifically, about the difference between Speedplay and Look pedals. To narrow it down even further, about why most pro cyclists ride on Look if Speedplay is supposedly better. Decisions, decisions… Can be overwhelming, and can lead to paralysis by analysis.

But in the grand scheme of things these details are nearly irrelevant. The marginal (and often dubious) performance improvement that this or that component affords the rider is almost always dwarfed by other, much more decisive factors: training, tactics, road conditions, training, training. The pros ride on whatever they ride because this is what they are being paid to ride. Their job is to RIDE THE FRIGGIN’ BIKE, so they focus on that, not on the pedals.

What does that have to do with GIS? For me, a lot.

I have been thinking a lot about GIS tools lately. Is QGIS better than ArcGIS? Is PostGIS better than file geodatabase? Is the shapefile the vinyl of geo? Should I keep my data in PostGIS and edit in QGIS, or should I use GISquirrel and edit PostGIS in ArcMap, or should I get Server and SDE and do it all in MSSQL? Decisions, decisions… Can be overwhelming, and can lead to paralysis by analysis.

But in the grand scheme of things these details are nearly irrelevant. The marginal (and often dubious) performance improvement that this or that GIS tool affords the GIS pro is almost always dwarfed by other, much more decisive factors: experience, training, environment, experience, experience. In the end it is important to remember that I am being paid TO DO THE FRIGGIN’ GIS, so I should focus on that, not on the tools.

So I will.

***

Ultimately, I resolve to think more about bikes and less about GIS. Speaking of which, what is the best lube for Speedplay cleats?