When you reach a trident in the road, take it

Many thanks to Bill Dollins, Craig Williams, and Randal Hale, whose generous counsel and thoughtful insights continue to guide me along my GIS journey.

I am facing the desktop GIS trident: ArcMap, QGIS, ArcGIS Pro. Three tines. Which one should I take?

In my earlier cycling metaphor I compared selecting bicycle pedals to selecting GIS software tools. The pedals debate is over (Speedplay won); the GIS debate still rages on. I have analyzed and overanalyzed. I have consulted with the wisdom of crowds (Twitter) and the wisdom of individuals (Twitter DMs). I have considered various platforms, data formats, and multiple combinations and permutations thereof. It is time to wrap it up.

“I hear and I forget. I see and I remember. I do and I understand.” –Confucius

I begin with a 30-day challenge: use nothing but ArcGIS Pro for all my day-to-day GIS activities. I will document my experiences — good and bad — and I will update this post continuously with a running tally of said experiences. I will sum it all up on March 15.

Here we go.

  • DAY ONE [1], February 14, 2019: The challenge begins. QGIS and ArcMap and ArcCatalog icons stare at me from the Windows 10 taskbar. I do not click on them, which feels weird. I click on the ArcGIS Pro icon next to them. I am greeted with a warning: “Pro Advanced will expire in 14 days.” Stress levels go up. We always pay our bills, but sometimes we are late. What will happen on March 1 if our P.O. doesn’t make it to Redlands in time? ArcMap and QGIS never give me that kind of stress. At least all my data is on our local network. I click OK. ArcGIS Pro 2.3 launches. I single-click on a recent project. Project loads. I see my map. Some of my data is in a file geodatabase (FGDB), some in shapefiles (SHP). At some point I will need to replicate a Desktop process to periodically export a FGDB feature class to SHP and manipulate said SHP (delete and add fields) before I upload the SHP into a proprietary system. I’ll also need to replicate a separate process where I join an SHP to an Access database. I’ll attempt all of that next week. Today I’ll just tackle some light tasks. I create a custom basemap from custom aerial photography. It works. I merge two polygons. The process is smooth, similar to ArcMap’s. I hit the Save button. Does it save the project, or the merged polygons?
  • DAY TWO [2], February 15, 2019: “Pro Advanced will expire in 13 days.” Okay then. I open the same project. My custom base map is gone, even though before shutting down yesterday I clicked every Save button I could find. What did I miss? OK, it’s right there in the documentation — the custom base map has to be saved in the project you want to use it in. You can’t share custom base maps across projects. I recreate the same custom basemap in the current project. On to Catalog. Add Folder Connection works just like on Desktop. It does not auto-refresh, so if you add a new file to a connected folder you won’t see the file until you hit Refresh. Deleting fields in SHP took considerably longer than the same process in Desktop, but not a big deal for me. Adding fields is similar to Desktop, while the ability to add more than one field at a time is a welcome improvement. Now let’s join this puppy to an MS Access table. Wait, what? No MS Access support in Pro? I found a discussion on an Esri support site, basically saying don’t use Access. I beg to differ. One of my workflows is constrained by hard external factors, and one of these factors is “use Access”. Another is “use SHP”. So I will continue to use Access and SHP. But I should be able to find a workaround. Export Access table do DBF doesn’t work because long field names get truncated, effectively resulting in trying to have the same field name more than once in the same table, which DBF doesn’t like. Export to Excel and then join SHP to XLSX works. Whew! The world’s most popular database saves the day.
  • DAY THREE [6], February 19, 2019: “Pro Advanced will expire in 9 days.” Got it. I click OK. ArcGIS Pro crashes. Should I send a report to Esri? I don’t always do, but I decide to send a report this time. Report sent. Launching Pro again. “Pro Advanced will expire in 9 days.” OK. Pro runs. First I address a blog comment question from Nancy von Meyer: “Can you make a starting project with your custom base map and save that with just the base map then open it and save as something different and have the custom base map in the saved as project?” I try this. Yes, you can, and this is a convenient way to save and reuse your custom basemap. Thank you for the suggestion, Nancy! Next I decide to tackle a monthly GDB maintenance task — “Compact”. (There is also “compress”; find the difference between compact and compress on your own.) It is unclear whether it’s safe to run Compact while having the same GDB in a map in the same project. I decide to run the process on a scratch GDB and see if something blows up. Compact runs, nothing blows up. I reindex a feature class in the scratch GDB. Nothing blows up. Save project. Close Pro. Open Pro. All my stuff is still there — custom base map, recently-compacted and reindexed scratch GDB, production GDB. On to compact and reindex the production GDB. Smooth sailing. (In recent weeks I have been getting error messages while editing and saving this same GDB in ArcMap. Errors were blamed (incorrectly, IMHO) on bad network connection. Very curious to see whether the same errors on the same GDB will crop up in Pro. So far they have not. Good!)
  • DAY FOUR [7], February 20, 2019: Day of routine tasks, already written up in prior days. No surprises — good or bad. Pro just works.
  • DAY FIVE [8], February 21, 2019: Annotation. Everyone’s got a pressure point, and mine is annotation. I know — not sexy, unhip, lame. Nothing like 3D or a billion points in a database in the cloud. But annotation is what brought me to the GIS trident. Specifically, the need to update and maintain going forward a fairly complex map for print, some of whose sources are personal GDB annotation feature classes (don’t ask). The map — some of whose layers I have inherited and have built upon — gives me stress every time I touch it. Will something blow up? I hope that moving everything to a fresh platform will make this stress go away, and the transition will have been worth it — both technically and psychologically. But do I move (or recreate) these feature classes to ArcMap, or QGIS, or Pro? Hence the GIS trident. Hence this crucial point in the 30-day ArcGIS Pro challenge, the acid test: Will Pro and its annotation capabilities rise to the occasion? Let’s find out. Add road centerline layer to map; zoom to desired scale (determined by trial and error with previous map); label roads; play with symbols and position. Everything works as expected, although redraws take a tad longer than I am used to. I can live with that. Convert To Annotation. “Create feature-linked annotation” shows a warning — option is only available with ArcGIS Desktop Standard and ArcGIS Desktop Advanced licenses. Clearly Pro is not separate from Desktop. I do have a Desktop Advanced license, so I proceed to run the process. Error. Feature-linked annotation can only be created in the same geodatabase as the base features. Repointing target destination. Run. Same error, even though I redirected the output anno to the same GDB as the base. Save everything. Quit Pro. Start Pro. Run Convert to Annotation again. Process fails with the same error. Oh, well… Nobody said it’d be easy. Easy is for amateurs. Could it be that even though I have Desktop Advanced, Pro doesn’t KNOW that I have Desktop Advanced because the two are licensed differently and don’t talk to each other? I’ll tackle that tomorrow.
  • DAY SIX [9], February 22, 2019: Annotation, day two. We have just paid our Esri bill and the licensing expiration warning message is gone. Could this have caused the anno error? I uncheck the “Create feature-linked annotation” and hit Run. The process runs. I check it — the process fails. What else could be causing the problem? The level of my license, or something else? Craig Williams offered several suggestions (thank you, Craig!), and one of them resolved the issue. It turns out that “Create feature-linked annotation” attempts to create annotation from all labels in the map. As it were, I had in the same map a shapefile-based layer, labelled, which was the culprit. As soon as I turned off the shapefile layer and its labels, Pro created feature-linked annotation from my intended road centerline layer’s labels. Pro is very close to passing the acid test. Next step in the anno test: manually edit annotation. I’ll tackle this next week.
  • DAY SEVEN [12], February 25, 2019: An unscheduled task takes precedence over the planned anno editing: Georeference a scanned image and digitize over it. Georeferencing and subsequent polygon digitization and attribute table editing are smooth and straightforward. BTW, the attribute editing on this same FGDB is where ArcMap would give me the occasional weird error. No errors in Pro so far. However, it is not always clear what I am selecting (I inadvertently deleted a layer from my map instead of a polygon from said layer), but I’m sure I’ll get the hang of it with practice. To resolve the inadvertent layer deletion (no “undo”), I opted for quitting Pro without saving the most recent action. Pro took a VERY long time to close the project and quit. After reopening Pro I open the same project. All my stuff is still there. Whew! I was saving the following comment for the summary at the end of the challenge, but I’ll jump the gun and say it now. I understand why Esri needs to sunset ArcMap. I appreciate all the new development that went and continues to go into Pro. But sometimes I wish Esri would put all the new Pro tech behind the familiar workflows of ArcMap, split the Pro product into Map and Cloud, and give the map-makers and cloudsters all the tools each group needs and none of what they don’t. Because, to quote myself: “The intersect of #ArcGISPro users who make maps for print and those who need billions of points in a database in the cloud is NULL.”
  • DAY EIGHT [13], February 26, 2019: Got a request to create a shapefile (“Ask for it by name!”) from a text file with coordinates. Pro delivers — creates a FGDB feature class first, which I then export to shapefile to satisfy the SHP delivery format request. Another unscheduled task takes precedence over the planned anno editing: Update one in a series of 30 PDF (of MXD lineage) maps. The map update itself should be trivial, but this will present a less trivial logistical / data management dilemma: What to do with the rest of the maps in the set? Don’t touch them and end up with a hybrid (MXD / APRX) map set, or convert them all to APRX and effectively predetermine the 30-day challenge outcome? I decide to cheat and update the map in ArcMap — not because Pro can’t handle the task, but to not contaminate the outcome of the 30-day challenge. I violate the rules of the challenge in order to keep it pure.
  • DAY NINE [14], February 27, 2019: Day of routine tasks, already chronicled in previous days. Pro just works, I am getting the hang of it. The crashes are gone, as is the license expiration warning (causality between crashes and impending license expiration unclear). No FGDB save errors, either. Overall a very smooth experience, if you don’t count the hunting and pecking for functions you know are there but who are seemingly playing hide-and-seek with you.
  • DAY TEN [15], February 28, 2019: Another day of routine tasks, so I’ll pontificate about credits consumption. Last year I inadvertently and accidentally (and quite quickly and easily) consumed a large number of service credits, just by clicking the wrong button two or maybe three times (Esri acknowledged my mistake and graciously refunded the account). Ever since I have been keeping a daily eye on the account balance, to make sure I have not possibly accidentally clicked on that (or a similarly-credits-draining) button again. The credits price list is long and confusing. It is mostly clear and makes sense while I read it, but a few days or weeks later doubts inevitably creep up (“Does this geoprocessing button consume credits?”) and I have to pause and check the price list again. It would be very nice if Pro unambiguously, clearly, and loudly informed users about credits consumption — up to and including putting the free geoprocessing tools behind a completely different section of the ribbon than credit-consuming functions.
  • DAY ELEVEN [16], March 1, 2019: Midway upon the journey of my 30-day ArcGIS Pro challenge I remind myself that the purpose of the challenge is to test whether I can replace Map with Pro. Not a comprehensive review of Pro’s capabilities. To put it bluntly: Can I stop using Map and still do my job. That is all. To that end I will not be testing integration with ArcGIS Online (AGOL), for this is not a Map function I’ll need to replace. Apropos, I have been using Pro to author online maps since 2016. Seven of these maps are shared with the public in an online portal. The process is not without its quirks, but workable. Credit consumption is minimal (over the course of one year I have consumed 24.01% of the complimentary credits that come with the annual  Esri maintenance). But enough about AGOL. Back to the daily grind where an internal customer calls for help with creating a list (in Excel) of property owners based on a fairly complex spatial selection. A breeze for Pro up until exporting the selected attribute data record set, where copy and paste into Excel just works, except Excel truncates the leading zeros in the zip code column (this is NJ, y’all), so I end up using the Excel CONCATENATE function to tack on a zero in front of each now-four-digit ex-zip code. On second thought, I could have pre-configured the receiving Excel spreadsheet column as text. I’ll try to remember next time. On that note, have a nice weekend, and see you Monday!
  • DAY TWELVE [19], March 4, 2019: Once per month I take a snapshot of my GIS data and upload it to a proprietary application accessed by many users. While I am free to choose the format in which I store and maintain the master GIS dataset (FGDB, PostGIS, coverage (I would if I could), whatever), the monthly snapshot requires SHP and Access, non-negotiable. Today and tomorrow (it’s a two-day process) will tell whether Pro can replace Map in this vital step. First (minor) hiccup: Unlike Map, Pro won’t write a shapefile with dashes in the filename. Dashes are useful if I want to include an ISO-formatted date in the filename, like so: “Description_2019-03-04.shp”. Should I eliminate the dashes or replace them with underscores? I opt for underscores over the ISO-compliant “20190304”. The process runs in the Geoprocessing panel, causing a brief blood pressure spike. “Does this consume service credits?” The new shapefile is automatically added to my map, want it or not. Next I need to delete some fields from the new shapefile’s attribute table. I should probably remove the layer from the map first (old skool lessons learned). I do that, and the Drop Field process runs successfully. Would it have run with the layer still in the map? I’ll try that some other time. One difference from the Delete Field process in ArcMap/Catalog is that in the Drop Field panel Pro rearranges the field names alphabetically — uncalled for and confusing, IMHO. Next I add several new fields to the SHP — the process is smooth and straightforward. So far so good.
  • DAY THIRTEEN [20], March 5, 2019: Import SHP into FGDB feature class. This is a big step, psychologically. Because this SHP is an export from a QGIS layer living in PostGIS. And that layer ended up in PostGIS because ArcMap had been giving me random, inexplicable errors on saving in its previous home — an FGDB. Which is one of several reasons that brought me to the trident — the ArcMap/FGDB combo had become unreliable, so I sought refuge in PostGIS (edited alternately by QGIS and GISquirrel on top or ArcMap). However, at this point in the 30-day challenge, I feel confident enough to bring my precious data back into an FGDB, this time managed in Pro. The ultimate goal being that, as Einstein allegedly said, “Everything should be made as simple as possible, but not simpler.” On a different note, tomorrow I will do my best to join a new SHP to a new Access table, as part two of that two-day process. Stay tuned.
  • DAY FOURTEEN [21], March 6, 2019: Part two of the abovementioned two-day process. Export SHP and Access from proprietary app, add SHP to Pro map, export Access table to Excel, add Excel table to Pro map, join SHP and Excel, import newly-joined SHP into FGDB feature class, done! Check for no-matches: Select by attribute where ‘fieldname’ is null. Bingo! Very pleased with the whole process. This was a major step in the 30-day challenge, and a deciding factor in whether Pro will get the thumbs up from this critic. Next, another export FGDB -> SHP -> ZIP to share with data partners (who ask for SHP by name, FTR). A noticeable difference — the resulting ZIP file size is 39% larger than that of a similarly-processed and -compiled file using ArcMap. Moving right along. Once again, just like I did on DAY SEVEN, I mistakenly delete a layer from the map instead of deleting a single selected feature in that layer. Am I obtuse? Also (indirectly related) I can select a feature and delete it while in the Map tab, but to save the edit I must first click on the Edit tab in order to expose the Save button which is otherwise hidden. Very convoluted.
  • DAY FIFTEEN [22], March 7, 2019: Editing in Pro all day. Lots of improved tools. I love it. No complaints today. I have been twitter-following the Esri Developer Summit, and I see all these new exciting things, and I see all these users tweeting excitedly about all these new exciting things. And I want to tell them: GIS is a marathon, not a sprint. By the time you get around to implementing these exciting new things, many of them will be obsolete or discontinued. Play the long game. Don’t turn up your nose on innovation, but don’t turn your back on the shapefile, either.
  • DAY SIXTEEN [23], March 8, 2019: Import MXD into Pro = Insert Map (MXD) into current project. Pro imports the Layout View, true to the original. Export to PDF (Share Layout) works as expected. 80% of GIS functionality covered right there. Or possibly 90%. I wonder if I can import the ArcMap Data View. Yes! Apparently Pro imports the state of the MXD in which it was last saved. Save the Layout View — import the layout. Save the Data View — import the “Map”. Am I ready for the Map-to-Pro transition, one week early? Possibly, but no reason to rush. Hang around for one more week, and read a summary of the 30-day challenge next Friday, March 15, 2019. Have a nice weekend!
  • DAY SEVENTEEN [26], March 11, 2019: Today I intended to import and edit FGDB annotation and Personal GDB annotation from ArcMap. Then I realized that I have become quite comfortable with labeling and creating feature-linked annotation in Pro. The much improved label placement algorithms and extensive set of user controls for labeling leave very little need to edit the new annotation. I decide that it would be more efficient to recreate all needed annotation in Pro than to try to salvage the existing ArcMap FGDB and personal GDB annotation.
  • DAY EIGHTEEN [27], March 12, 2019: Will an ArcMap custom tool work in Pro? In ArcMap I have been using a Python (ArcPy) script (Desktop ArcToolbox tool) that selects streets that intersect with grid, creates a street index, and exports to PDF. A nice, dressed-up spatial join. I found this tool (developed by Wes Miller) years ago on an Esri support page that no longer exists. Will this code work in Pro (with or without tweaks)? I drag and drop the TBX into my Tools folder. I launch the tool. I fill in all required parameters, the tool runs for a while, then throws an error, understandably. Line 52: “mxd = arcpy.mapping.MapDocument(mxdDoc)”. Should I tweak the code to make it work in Pro? On brief reflection, I decide to just run a straight spatial join. I do, it gives me the output I need. Stay as close to the core as possible. One less moving part. I congratulate myself for simplifying my life.
  • DAY NINETEEN [28], March 13, 2019: As the 30-day challenge winds down, I am running out of new things to try. While I am sure new types of tasks will present themselves in due course, for now I have covered pretty much all aspects of my regular workflow. I realize that there are vast areas of Pro functionality that I haven’t touched –3D, raster analysis, network analysis, etc. I will tackle those if and when the need arises. Presently I feel like this prospective cell phone customer: do-you-have-a-phone-that-doesn-t-do-too-much-new-yorker-cartoon
  • DAY TWENTY [29], March 14, 2019: On this penultimate day of the 30-day ArcGIS Pro challenge I ruminate quietly on the ideology and practicality of software choice. I will leave off the long of it for a future tractate. The short of it is: Should I also do a 30-day QGIS challenge? After much thought I decide that this would only be necessary if I determine that Pro can not fully replace Map in my daily duties. The trident described in the beginning of this post is reduced to a bident, with QGIS lying in wait for Pro to fail. Will Pro fail? Tune in tomorrow for the 30-day ArcGIS Pro challenge summary.
  • DAY TWENTY ONE [30], March 15, 2019: The 30-day ArcGIS Pro challenge ends.

What did I learn?

TL;DR

The 30-day ArcGIS Pro challenge was designed to answer one question: Can I replace ArcMap (Advanced) with ArcGIS Pro (Advanced) and still do my job? The answer is yes.

FACTS

  • ArcGIS Pro works.
  • As of today (March 15, 2019) Pro has all the functionality I need to do my daily job.
  • Pro is being developed actively.
  • Pro refreshes the OSM basemap practically instantly
  • The imagery basemap (out beyond ~3,000 scale) is presently the most up-to-date basemap out of the four basemaps I use across all applications
  • Pro crashes occasionally on launch (rarely).
  • Pro doesn’t support MS Access.

OPINIONS

The Pro crashes don’t bother me (as long as Pro doesn’t take down my data with it, which it does not). I trust that the crashes will go away as the product matures and stabilizes.

ArcGIS Pro is complex. Unnecessarily complex, for my needs. At times I feel like I am driving an 18-wheeler to the corner deli to get a sandwich. A four-wheeler will do, or even a two-wheeler. I wish Esri would have split Pro into the equivalents of 18-wheeler, pickup truck, passenger car. Or ProMap for map production, ProCloud, Pro3D…

Pro is confusing. Probably less so for someone just getting her feet wet in GIS, but definitely for someone used to the ways of ArcMap and ArcInfo Workstation and PC ARC/INFO and ArcView 3 and ArcView 2 and ArcView 1. Many times during this trial I wished Esri had put the new technology developed for Pro behind the familiar workflow of ArcMap. Not the interface — I don’t mind a new interface — but the workflows ingrained in any seasoned geogeek’s subconscious.

WISH LIST

MUST HAVE:

  • MS Access support
  • Save Edits button visible any time an edit function is performed

NICE TO HAVE:

  • Different symbols/icons in the table of contents depending on source type, like in Catalog (SHP vs FGDB, etc.)
  • Recalculate map extent as new layers are added or removed.
  • Make the Export function more user-friendly. Currently I can export to SHP or DBF, but I must type the file extension to make it happen. Not really intuitive. By default all exports go into FGDB. It would be nice to have a drop-down to specify SHP, DBF, etc.

TIPS

  • Compact your FGDB(s) monthly
  • Reindex your FGDB feature classes monthly (Recalculate Spatial Index)

SUMMARY

ArcGIS Pro has all the functionality I need to do my daily job. I am making Pro my default GIS tool, effective immediately. Over the course of the last month (21 business days) I have grown familiar with and even fond of Pro. However, the full transition from Map to Pro will not be instantaneous. I will not be uninstalling ArcMap from my PC anytime soon.

Pro gave me the confidence (and nudge) to clean up my data and streamline my workflows. It prompted me to ask (and answer) questions like: How much of the past do I need to preserve? What is important and what is not? Obviously I want to preserve my core data, like road centerlines. But what about derivative data, e.g., annotation derived from those centerlines? Knowing that some of those annotations have been tortured into submission to fit into their allotted spaces, is it reasonable to expect that they will move over smoothly to the new platform? Does it not make more sense to recreate them from scratch? Ditto for some cosmetic customizations, which are nice to have but not crucial for my workflow — drop them and stay close to the core?

Pro taught me humility. I jumped into it, cocky, flaunting my 28 years of GIS experience. I stumbled many times. I learned from my mistakes, sometimes while cursing.

It is a virtual certainty that new types of GIS tasks will come my way in the future — tasks I did not have the opportunity to test during the 30-day challenge. I am reasonably confident that I will be able to handle them in Pro, and I look forward to the challenge.

Finally, what about QGIS? Is it not a great tool? I have been using QGIS for many years. I have come to appreciate its growing functionality, the effort of its developers, and the concept of freedom that it espouses. I am well aware that QGIS outperforms ArcMap / Pro in some areas. I am also aware that there are multiple factors into measuring performance, workflow compatibility being a major one. At my current station in life and career, embracing QGIS as my primary tool is just not practical.

Thank you for reading thus far. I hope this write-up has been useful. Have questions? Leave a comment, or hit me up on Twitter, or email me at atanas [at] entchev.com.